table of contentsdatacenter to datacenter replication on kubernetes highlights scalability...

1.1

1.2

1.2.1

1.2.1.1

1.2.1.2

1.2.1.3

1.2.1.4

1.2.2

1.2.3

1.2.4

1.3

1.3.1

1.3.2

1.4

1.5

1.5.1

1.5.2

1.5.3

1.6

1.6.1

1.6.2

1.6.2.1

1.6.2.2

1.6.3

1.6.3.1

1.6.3.2

1.6.4

1.6.4.1

1.6.4.2

1.6.4.3

1.6.5

1.6.6

1.6.6.1

1.6.6.2

1.6.6.3

1.6.6.4

1.7

1.7.1

1.7.2

1.7.3

1.7.4

TableofContentsIntroduction

GettingStarted

Installing

Linux

MacOSX

Windows

Compiling

Authentication

AccessingtheWebInterface

ComingfromSQL

Tutorials

Kubernetes

DatacentertodatacenterReplicationonKubernetes

Highlights

Scalability

Architecture

Datamodels

Limitations

Datamodels&modeling

Concepts

Databases

WorkingwithDatabases

NotesaboutDatabases

Collections

CollectionMethods

DatabaseMethods

Documents

BasicsandTerminology

CollectionMethods

DatabaseMethods

Graphs,Vertices&Edges

NamingConventions

DatabaseNames

CollectionNames

DocumentKeys

AttributeNames

Indexing

IndexBasics

Whichindextousewhen

IndexUtilization

WorkingwithIndexes

1

1.7.4.1

1.7.4.2

1.7.4.3

1.7.4.4

1.7.4.5

1.7.4.6

1.8

1.8.1

1.8.1.1

1.8.1.2

1.8.2

1.8.2.1

1.8.3

1.8.3.1

1.8.3.2

1.8.4

1.8.5

1.9

1.9.1

1.9.2

1.9.3

1.9.4

1.9.5

1.9.6

1.9.7

1.9.7.1

1.9.7.2

1.9.7.3

1.9.7.4

1.9.8

1.9.9

1.9.9.1

1.9.9.1.1

1.9.9.1.2

1.9.9.2

1.9.9.2.1

1.9.9.2.2

1.9.10

1.9.11

1.9.12

1.9.13

1.9.14

1.9.14.1

1.9.14.2

HashIndexes

Skiplists

Persistent

FulltextIndexes

GeoIndexes

VertexCentricIndexes

Graphs

GeneralGraphs

GraphManagement

GraphFunctions

SmartGraphs

SmartGraphManagement

Traversals

UsingTraversalObjects

ExampleData

WorkingwithEdges

Pregel

FoxxMicroservices

Ataglance

Gettingstarted

Servicemanifest

Servicecontext

Configuration

Dependencies

Routers

Endpoints

Middleware

Request

Response

UsingGraphQL

Sessionsmiddleware

Sessionstorages

Collectionstorage

JWTstorage

Sessiontransports

Cookietransport

Headertransport

Servingfiles

Writingtests

CrossOrigin

Scriptsandqueuedjobs

Migrating2.xservices

Migratingfrompre-2.8

manifest.json

2

1.9.14.3

1.9.14.4

1.9.14.5

1.9.14.5.1

1.9.14.5.2

1.9.14.5.3

1.9.14.5.4

1.9.14.5.5

1.9.14.5.6

1.9.14.6

1.9.14.7

1.9.14.8

1.9.15

1.9.16

1.9.17

1.9.17.1

1.9.17.2

1.9.17.3

1.10

1.10.1

1.10.2

1.10.3

1.10.4

1.10.5

1.11

1.11.1

1.11.2

1.11.2.1

1.11.2.2

1.11.2.3

1.11.2.3.1

1.11.2.3.2

1.11.2.3.3

1.11.2.3.4

1.11.3

1.11.4

1.11.4.1

1.11.4.2

1.11.4.3

1.11.4.4

1.11.4.5

1.11.4.6

1.11.4.7

1.11.4.8

applicationContext

RepositoriesandModels

Controllers

Requestcontext

Errorhandling

Before/After/Around

Requestobject

Responseobject

DependencyInjection

Sessions

AuthandOAuth2

FoxxQueries

Legacycompatibilitymode

Usermanagement

Relatedmodules

Authentication

OAuth1.0a

OAuth2.0

Transactions

Transactioninvocation

Passingparameters

Lockingandisolation

Durability

Limitations

Deployment

Singleinstance

Cluster

Mesos,DC/OS

Generic&Docker

AdvancedTopics

StandaloneAgency

Localtestsetups

Processes

Docker

MultipleDatacenters

Kubernetes

UsingtheOperator

DeploymentResourceReference

DriverConfiguration

Authentication

Scaling

Upgrading

ArangoDBConfiguration&Secrets

Metrics

3

1.11.4.9

1.11.4.10

1.11.4.11

1.11.4.12

1.11.4.13

1.11.4.14

1.12

1.12.1

1.12.1.1

1.12.1.2

1.12.1.3

1.12.1.4

1.12.1.5

1.12.1.6

1.12.1.7

1.12.1.8

1.12.1.9

1.12.2

1.12.2.1

1.12.2.2

1.12.2.3

1.12.3

1.12.4

1.12.5

1.12.6

1.12.7

1.12.7.1

1.12.8

1.12.8.1

1.12.8.2

1.12.8.3

1.12.8.4

1.12.8.5

1.12.8.6

1.12.8.7

1.12.8.8

1.12.8.9

1.12.8.10

1.12.8.11

1.12.8.12

1.12.9

1.12.10

1.12.11

1.12.11.1

Services&Loadbalancer

DeploymentReplicationResourceReference

Storage

StorageResource

TLS

Troubleshooting

Administration

WebInterface

Dashboard

Cluster

Collections

Document

Queries

Graphs

Services

Users

Logs

ArangoDBShell

ShellOutput

Configuration

Details

Arangoimp

Arangodump

Arangorestore

Arangoexport

ManagingUsers

InArangosh

ServerConfiguration

OperatingSystemConfiguration

ManagingEndpoints

SSLConfiguration

LDAPOptions

LoggingOptions

GeneralOptions

Write-AheadLogOptions

CompactionOptions

ClusterOptions

RocksDBEngineOptions

HashCacheOptions

AsynchronousTasks

Durability

Encryption

Auditing

Configuration

4

1.12.11.2

1.12.12

1.12.12.1

1.12.12.1.1

1.12.12.1.2

1.12.12.1.3

1.12.12.1.4

1.12.12.1.5

1.12.12.2

1.12.12.2.1

1.12.12.2.2

1.12.12.3

1.12.13

1.12.14

1.12.14.1

1.12.14.2

1.12.14.3

1.12.14.4

1.12.14.5

1.12.14.6

1.12.14.7

1.12.14.8

1.12.14.9

1.12.14.10

1.13

1.13.1

1.13.2

1.13.3

1.13.4

1.14

1.14.1

1.14.2

1.15

1.15.1

1.15.2

1.15.3

1.15.4

1.15.5

1.15.6

1.15.7

1.15.8

1.15.9

1.15.10

1.15.11

Events

Replication

AsynchronousReplication

Components

Per-DatabaseSetup

Server-LevelSetup

SyncingCollections

ReplicationLimitations

SynchronousReplication

Implementation

Configuration

SatelliteCollections

Sharding

Upgrading

Upgradingto3.3

Upgradingto3.2

Upgradingto3.1

Upgradingto3.0

Upgradingto2.8

Upgradingto2.6

Upgradingto2.5

Upgradingto2.4

Upgradingto2.3

Upgradingto2.2

Troubleshooting

arangod

EmergencyConsole

DatafileDebugger

Arangobench

Architecture

Write-aheadlog

StorageEngines

Releasenotes

WhatsNewin3.3

Incompatiblechangesin3.3

WhatsNewin3.2

KnownIssuesin3.2


WhatsNewin3.1


WhatsNewin3.0


WhatsNewin2.8


5

1.15.12

1.15.13

1.15.14

1.15.15

1.15.16

1.15.17

1.15.18

1.15.19

1.15.20

1.15.21

1.15.22

1.15.23

1.16

1.16.1

1.16.1.1

1.16.1.2

1.16.2

1.16.2.1

1.16.2.2

1.16.2.3

1.16.2.4

1.16.2.5

1.16.2.6

1.16.2.7

1.16.2.8

1.16.2.9

1.16.3

1.16.3.1

1.16.3.1.1

1.16.3.1.2

1.16.3.1.3

1.16.3.1.4

1.16.3.1.5

1.16.3.2

1.16.3.2.1

1.16.3.2.2

1.16.3.2.3

1.16.4

1.16.5

WhatsNewin2.7


WhatsNewin2.6


WhatsNewin2.5


WhatsNewin2.4


WhatsNewin2.3


WhatsNewin2.2

WhatsNewin2.1

Appendix

References

db

collection

JavaScriptModules

@arangodb

console

crypto

fs

request

actions

queries

Write-aheadlog

TaskManagement

Deprecated

SimpleQueries

SequentialAccess

Pagination

ModificationQueries

GeoQueries

FulltextQueries

Actions

DeliveringHTMLPages

JsonObjects

Modifying

Errorcodesandmeanings

Glossary

6

ArangoDBv3.3.10DocumentationWelcometotheArangoDBdocumentation!

NewandeagertotryoutArangoDB?Startrightawaywithourbeginner'sguide:GettingStarted

Thedocumentationisorganizedinfourhandbooks:

ThismanualdescribesArangoDBanditsfeaturesindetailforyouasauser,developerandadministrator.TheAQLhandbookexplainsArangoDB'squerylanguageAQL.TheHTTPhandbookdescribestheinternalAPIofArangoDBthatisusedtocommunicatewithclients.Ingeneral,theHTTPhandbookwillbeofinteresttodriverdevelopers.Ifyouuseanyoftheexistingdriversforthelanguageofyourchoice,youcanskipthishandbook.Ourcookbookwithrecipesforspecificproblemsandsolutions.

Featuresareillustratedwithinteractiveusageexamples;youcancut'n'pastethemintoarangoshtotrythemout.TheHTTPREST-APIfordriverdevelopersisdemonstratedwithcut'n'pasterecipesintendedtobeusedwiththecURL.Driversmayprovidetheirownexamplesbasedonthese.jsbasedexamplestoimproveunderstandabilityfortheirrespectiveusers,i.e.forthejavadriversomeofthesamplesarere-implemented.

Overview

ArangoDBisanativemulti-model,open-sourcedatabasewithflexibledatamodelsfordocuments,graphs,andkey-values.BuildhighperformanceapplicationsusingaconvenientSQL-likequerylanguageorJavaScriptextensions.UseACIDtransactionsifyourequirethem.Scalehorizontallyandverticallywithafewmouseclicks.

Keyfeaturesinclude:

installingArangoDBonaclusterisaseasyasinstallinganapponyourmobileFlexibledatamodeling:modelyourdataascombinationofkey-valuepairs,documentsorgraphs-perfectforsocialrelationsPowerfulquerylanguage(AQL)toretrieveandmodifydataUseArangoDBasanapplicationserverandfuseyourapplicationanddatabasetogetherformaximalthroughputTransactions:runqueriesonmultipledocumentsorcollectionswithoptionaltransactionalconsistencyandisolationReplicationandSharding:setupthedatabaseinamaster-slaveconfigurationorspreadbiggerdatasetsacrossmultipleserversConfigurabledurability:lettheapplicationdecideifitneedsmoredurabilityormoreperformanceNo-nonsensestorage:ArangoDBusesallofthepowerofmodernstoragehardware,likeSSDandlargecachesJavaScriptforall:nolanguagezoo,youcanuseonelanguagefromyourbrowsertoyourback-endArangoDBcanbeeasilydeployedasafault-tolerantdistributedstatemachine,whichcanserveastheanimalbrainofdistributedappliancesItisopensource(ApacheLicense2.0)

Community

IfyouhavequestionsregardingArangoDB,Foxx,drivers,orthisdocumentationdon'thesitatetocontactuson:

GitHubforissuesandmisbehaviororpullrequestsGoogleGroupsfordiscussionsaboutArangoDBingeneralortoannounceyournewFoxxAppStackOverflowforquestionsaboutAQL,usagescenariosetc.Slack,ourcommunitychat

Whenreportingissues,pleasedescribe:

theenvironmentyourunArangoDBintheArangoDBversionyouusewhetheryou'reusingFoxxtheclientyou'reusing

Introduction

7

http://curl.haxx.se

https://github.com/arangodb/arangodb-java-driver#learn-more

https://github.com/arangodb/arangodb/issues

https://www.arangodb.com/community/

https://groups.google.com/forum/?hl=de#!forum/arangodb

http://stackoverflow.com/questions/tagged/arangodb

http://slack.arangodb.com

whichpartsofthedocumentationyou'reworkingwith(link)whatyouexpecttohappenwhatisactuallyhappening

Wewillrespondassoonaspossible.

Introduction

8

Gettingstarted

Overview

Thisbeginner'sguidewillmakeyoufamiliarwithArangoDB.Wewillcoverhowto

installandrunalocalArangoDBserverusethewebinterfacetointeractwithitstoreexampledatainthedatabasequerythedatabasetoretrievethedataagaineditandremoveexistingdata

Installation

Headtoarangodb.com/download,selectyouroperatingsystemanddownloadArangoDB.Youmayalsofollowtheinstructionsonhowtoinstallwithapackagemanager,ifavailable.

IfyouinstalledabinarypackageunderLinux,theserverisautomaticallystarted.

IfyouinstalledArangoDBusinghomebrewunderMacOSX,starttheserverbyrunning/usr/local/sbin/arangod.

IfyouinstalledArangoDBunderWindowsasaservice,theserverisautomaticallystarted.Otherwise,runthearangod.exelocatedintheinstallationfolder'sbindirectory.YoumayhavetorunitasadministratortograntitwritepermissionstoC:\ProgramFiles.

Formorein-depthinformationonhowtoinstallArangoDB,aswellasavailablestartupparameters,installationinaclusterandsoon,seeInstalling.

ArangoDBofferstwostorageengines:MMFilesandRocksDB.Choosetheonewhichsuitsyourneedsbestintheinstallationprocessoronfirststartup.

Securingtheinstallation

Thedefaultinstallationcontainsonedatabase_systemandausernamedroot.

DebianbasedpackagesandtheWindowsinstallerwillaskforapasswordduringtheinstallationprocess.Red-Hatbasedpackageswillsetarandompassword.Forallotherinstallationpackagesyouneedtoexecute

shell>arango-secure-installation

Thiswillaskforarootpasswordandsetsthispassword.

Webinterface

Theserveritself(arangod)speaksHTTP/REST,butyoucanusethegraphicalwebinterfacetokeepitsimple.There'salsoarangosh,asynchronousshellforinteractionwiththeserver.Ifyou'readeveloper,youmightprefertheshellovertheGUI.Itdoesnotprovidefeatureslikesyntaxhighlightinghowever.

WhenyoustartusingArangoDBinyourproject,youwilllikelyuseanofficialorcommunity-madedriverwritteninthesamelanguageasyourproject.Driversimplementaprogramminginterfacethatshouldfeelnaturalforthatprogramminglanguage,anddoallthetalkingtotheserver.Therefore,youcanmostcertainlyignoretheHTTPAPIunlessyouwanttowriteadriveryourselforexplicitlywanttousetherawinterface.

Togetfamiliarwiththedatabasesystemyoucanevenputdriversasideandusethewebinterface(codenameAardvark)forbasicinteraction.Thewebinterfacewillbecomeavailableshortlyafteryoustartedarangod.Youcanaccessitinyourbrowserathttp://localhost:8529-ifnot,pleaseseeTroubleshooting.

GettingStarted

9

https://www.arangodb.com/download/

http://localhost:8529

Bydefault,authenticationisenabled.Thedefaultuserisroot.Dependingontheinstallationmethodused,theinstallationprocesseitherpromptedfortherootpasswordorthedefaultrootpasswordisempty(seeabove).

Nextyouwillbeaskedwhichdatabasetouse.Everyserverinstancecomeswitha_systemdatabase.Selectthisdatabasetocontinue.

GettingStarted

10

Youshouldthenbepresentedthedashboardwithserverstatisticslikethis:

Foramoredetaileddescriptionoftheinterface,seeWebInterface.

Databases,collectionsanddocuments

Databasesaresetsofcollections.Collectionsstorerecords,whicharereferredtoasdocuments.CollectionsaretheequivalentoftablesinRDBMS,anddocumentscanbethoughtofasrowsinatable.Thedifferenceisthatyoudon'tdefinewhatcolumns(orratherattributes)therewillbeinadvance.Everydocumentinanycollectioncanhavearbitraryattributekeysandvalues.Documentsinasinglecollectionwilllikelyhaveasimilarstructureinpracticehowever,butthedatabasesystemitselfdoesnotimposeitandwilloperatestableandfastnomatterhowyourdatalookslike.

Readmoreinthedata-modelconceptschapter.

Fornow,youcanstickwiththedefault_systemdatabaseandusethewebinterfacetocreatecollectionsanddocuments.StartbyclickingtheCOLLECTIONSmenuentry,thentheAddCollectiontile.Giveitaname,e.g.users,leavetheothersettingsunchanged(wewantittobeadocumentcollection)andSaveit.Anewtilelabeledusersshouldshowup,whichyoucanclicktoopen.

TherewillbeNodocumentsyet.Clickthegreencirclewiththewhiteplusontheright-handsidetocreateafirstdocumentinthiscollection.Adialogwillaskyoufora_key.YoucanleavethefieldblankandclickCreatetoletthedatabasesystemassignanautomaticallygenerated(unique)key.Notethatthe_keypropertyisimmutable,whichmeansyoucannotchangeitoncethedocumentiscreated.Whatyoucanuseasdocumentkeyisdescribedinthenamingconventions.

Anautomaticallygeneratedkeycouldbe"9883"(_keyisalwaysastring!),andthedocument_idwouldbe"users/9883"inthatcase.Asidefromafewsystemattributes,thereisnothinginthisdocumentyet.Let'saddacustomattributebyclickingtheicontotheleftof(emptyobject),thenAppend.Twoinputfieldswillbecomeavailable,FIELD(attributekey)andVALUE(attributevalue).Typenameaskeyandyournameasvalue.Appendanotherattribute,nameitageandsetittoyourage.ClickSavetopersistthechanges.IfyouclickonCollection:usersatthetopontheright-handsideoftheArangoDBlogo,thedocumentbrowserwillshowthedocumentsintheuserscollectionandyouwillseethedocumentyoujustcreatedinthelist.

Queryingthedatabase

GettingStarted

11

TimetoretrieveourdocumentusingAQL,ArangoDB'squerylanguage.Wecandirectlylookupthedocumentwecreatedviathe_id,buttherearealsootheroptions.ClicktheQUERIESmenuentrytobringupthequeryeditorandtypethefollowing(adjustthedocumentIDtomatchyourdocument):

RETURNDOCUMENT("users/9883")

ThenclickExecutetorunthequery.Theresultappearsbelowthequeryeditor:

[

{

"_key":"9883",

"_id":"users/9883",

"_rev":"9883",

"age":32,

"name":"JohnSmith"

}

]

Asyoucansee,theentiredocumentincludingthesystemattributesisreturned.DOCUMENT()isafunctiontoretrieveasingledocumentoralistofdocumentsofwhichyouknowthe_keysor_ids.Wereturntheresultofthefunctioncallasourqueryresult,whichisourdocumentinsideoftheresultarray(wecouldhavereturnedmorethanoneresultwithadifferentquery,butevenforasingledocumentasresult,westillgetanarrayatthetoplevel).

Thistypeofqueryiscalleddataaccessquery.Nodataiscreated,changedordeleted.Thereisanothertypeofquerycalleddatamodificationquery.Let'sinsertaseconddocumentusingamodificationquery:

INSERT{name:"KatieFoster",age:27}INTOusers

Thequeryisprettyself-explanatory:theINSERTkeywordtellsArangoDBthatwewanttoinsertsomething.Whattoinsert,adocumentwithtwoattributesinthiscase,followsnext.Thecurlybraces{}signifydocuments,orobjects.Whentalkingaboutrecordsinacollection,wecallthemdocuments.EncodedasJSON,wecallthemobjects.Objectscanalsobenested.Here'sanexample:

{

"name":{

"first":"Katie",

"last":"Foster"

}

}

INTOisamandatorypartofeveryINSERToperationandisfollowedbythecollectionnamethatwewanttostorethedocumentin.Notethattherearenoquotemarksaroundthecollectionname.

Ifyourunabovequery,therewillbeanemptyarrayasresultbecausewedidnotspecifywhattoreturnusingaRETURNkeyword.Itisoptionalinmodificationqueries,butmandatoryindataaccessqueries.EvenwithRETURN,thereturnvaluecanstillbeanemptyarray,e.g.ifthespecifieddocumentwasnotfound.Despitetheemptyresult,theabovequerystillcreatedanewuserdocument.Youcanverifythiswiththedocumentbrowser.

Let'saddanotheruser,butreturnthenewlycreateddocumentthistime:

INSERT{name:"JamesHendrix",age:69}INTOusers

RETURNNEW

NEWisapseudo-variable,whichreferstothedocumentcreatedbyINSERT.Theresultofthequerywilllooklikethis:

[

{

"_key":"10074",

"_id":"users/10074",

"_rev":"10074",

"age":69,

"name":"JamesHendrix"

}

]

GettingStarted

12

Nowthatwehave3usersinourcollection,howtoretrievethemallwithasinglequery?Thefollowingdoesnotwork:




TherecanonlybeasingleRETURNstatementhereandasyntaxerrorisraisedifyoutrytoexecuteit.TheDOCUMENT()functionoffersasecondarysignaturetospecifymultipledocumenthandles,sowecoulddo:

RETURNDOCUMENT(["users/9883","users/9915","users/10074"])

Anarraywiththe_idsofall3documentsispassedtothefunction.Arraysaredenotedbysquarebrackets[]andtheirelementsareseparatedbycommas.

Butwhatifweaddmoreusers?Wewouldhavetochangethequerytoretrievethenewlyaddedusersaswell.Allwewanttosaywithourqueryis:"Foreveryuserinthecollectionusers,returntheuserdocument".WecanformulatethiswithaFORloop:

FORuserINusers

RETURNuser

Itexpressestoiterateovereverydocumentinusersandtouseuserasvariablename,whichwecanusetorefertothecurrentuserdocument.Itcouldalsobecalleddoc,uorahuacatlguacamole,thisisuptoyou.Itisadvisabletouseashortandself-descriptivenamehowever.

Theloopbodytellsthesystemtoreturnthevalueofthevariableuser,whichisasingleuserdocument.Alluserdocumentsarereturnedthisway:

[

{

"_key":"9915",

"_id":"users/9915",

"_rev":"9915",

"age":27,

"name":"KatieFoster"

},

{

"_key":"9883",

"_id":"users/9883",

"_rev":"9883",

"age":32,

"name":"JohnSmith"

},

{

"_key":"10074",

"_id":"users/10074",

"_rev":"10074",

"age":69,

"name":"JamesHendrix"

}

]

Youmayhavenoticedthattheorderofthereturneddocumentsisnotnecessarilythesameastheywereinserted.Thereisnoorderguaranteedunlessyouexplicitlysortthem.WecanaddaSORToperationveryeasily:

FORuserINusers

SORTuser._key

RETURNuser

Thisdoesstillnotreturnthedesiredresult:James(10074)isreturnedbeforeJohn(9883)andKatie(9915).Thereasonisthatthe_keyattributeisastringinArangoDB,andnotanumber.Theindividualcharactersofthestringsarecompared.1islowerthan9andtheresultistherefore"correct".Ifwewantedtousethenumericalvalueofthe_keyattributesinstead,wecouldconvertthestringtoanumberanduseitforsorting.Therearesomeimplicationshowever.Wearebetteroffsortingsomethingelse.Howabouttheage,indescendingorder?

GettingStarted

13

FORuserINusers

SORTuser.ageDESC

RETURNuser

Theuserswillbereturnedinthefollowingorder:James(69),John(32),Katie(27).InsteadofDESCfordescendingorder,ASCcanbeusedforascendingorder.ASCisthedefaultthoughandcanbeomitted.

Wemightwanttolimittheresultsettoasubsetofusers,basedontheageattributeforexample.Let'sreturnusersolderthan30only:

FORuserINusers

FILTERuser.age>30

SORTuser.age

RETURNuser

ThiswillreturnJohnandJames(inthisorder).Katie'sageattributedoesnotfulfillthecriterion(greaterthan30),sheisonly27andthereforenotpartoftheresultset.Wecanmakeheragetoreturnheruserdocumentagain,usingamodificationquery:

UPDATE"9915"WITH{age:40}INusers

RETURNNEW

UPDATEallowstopartiallyeditanexistingdocument.ThereisalsoREPLACE,whichwouldremoveallattributes(exceptfor_keyand_id,whichremainthesame)andonlyaddthespecifiedones.UPDATEontheotherhandonlyreplacesthespecifiedattributesandkeepseverythingelseas-is.

TheUPDATEkeywordisfollowedbythedocumentkey(oradocument/objectwitha_keyattribute)toidentifywhattomodify.TheattributestoupdatearewrittenasobjectaftertheWITHkeyword.INdenotesinwhichcollectiontoperformthisoperationin,justlikeINTO(bothkeywordsareactuallyinterchangablehere).ThefulldocumentwiththechangesappliedisreturnedifweusetheNEWpseudo-variable:

[

{

"_key":"9915",

"_id":"users/9915",

"_rev":"12864",

"age":40,

"name":"KatieFoster"

}

IfweusedREPLACEinstead,thenameattributewouldbegone.WithUPDATE,theattributeiskept(thesamewouldapplytoadditionalattributesifwehadthem).

LetusrunourFILTERqueryagain,butonlyreturntheusernamesthistime:

FORuserINusers

FILTERuser.age>30

SORTuser.age

RETURNuser.name

Thiswillreturnthenamesofall3users:

[

"JohnSmith",

"KatieFoster",

"JamesHendrix"

]

Itiscalledaprojectionifonlyasubsetofattributesisreturned.Anotherkindofprojectionistochangethestructureoftheresults:

FORuserINusers

RETURN{userName:user.name,age:user.age}

GettingStarted

14

Thequerydefinestheoutputformatforeveryuserdocument.TheusernameisreturnedasuserNameinsteadofname,theagekeepstheattributekeyinthisexample:

[

{

"userName":"JamesHendrix",

"age":69

},

{

"userName":"JohnSmith",

"age":32

},

{

"userName":"KatieFoster",

"age":40

}

]

Itisalsopossibletocomputenewvalues:

FORuserINusers

RETURNCONCAT(user.name,"'sageis",user.age)

CONCAT()isafunctionthatcanjoinelementstogethertoastring.Weuseitheretoreturnastatementforeveryuser.Asyoucansee,theresultsetdoesnotalwayshavetobeanarrayofobjects:

[

"JamesHendrix'sageis69",

"JohnSmith'sageis32",

"KatieFoster'sageis40"

]

Nowlet'sdosomethingcrazy:foreverydocumentintheuserscollection,iterateoveralluserdocumentsagainandreturnuserpairs,e.g.JohnandKatie.Wecanusealoopinsidealoopforthistogetthecrossproduct(everypossiblecombinationofalluserrecords,3*3=9).Wedon'twantpairingslikeJohn+Johnhowever,solet'seliminatethemwithafiltercondition:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

RETURN[user1.name,user2.name]

Weget6pairings.PairslikeJames+JohnandJohn+Jamesarebasicallyredundant,butfairenough:

[

["JamesHendrix","JohnSmith"],

["JamesHendrix","KatieFoster"],

["JohnSmith","JamesHendrix"],

["JohnSmith","KatieFoster"],

["KatieFoster","JamesHendrix"],

["KatieFoster","JohnSmith"]

]

Wecouldcalculatethesumofbothagesandcomputesomethingnewthisway:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

RETURN{

pair:[user1.name,user2.name],

sumOfAges:user1.age+user2.age

}

WeintroduceanewattributesumOfAgesandaddupbothagesforthevalue:

[

GettingStarted

15

{

"pair":["JamesHendrix","JohnSmith"],

"sumOfAges":101

},

{

"pair":["JamesHendrix","KatieFoster"],

"sumOfAges":109

},

{

"pair":["JohnSmith","JamesHendrix"],

"sumOfAges":101

},

{

"pair":["JohnSmith","KatieFoster"],

"sumOfAges":72

},

{

"pair":["KatieFoster","JamesHendrix"],

"sumOfAges":109

},

{

"pair":["KatieFoster","JohnSmith"],

"sumOfAges":72

}

]

Ifwewantedtopost-filteronthenewattributetoonlyreturnpairswithasumlessthan100,weshoulddefineavariabletotemporarilystorethesum,sothatwecanuseitinaFILTERstatementaswellasintheRETURNstatement:

FORuser1INusers

FORuser2INusers

FILTERuser1!=user2

LETsumOfAges=user1.age+user2.age

FILTERsumOfAges<100

RETURN{

pair:[user1.name,user2.name],

sumOfAges:sumOfAges

}

TheLETkeywordisfollowedbythedesignatedvariablename(sumOfAges),thenthere'sa=symbolandthevalueoranexpressiontodefinewhatvaluethevariableissupposedtohave.Were-useourexpressiontocalculatethesumhere.WethenhaveanotherFILTERtoskiptheunwantedpairingsandmakeuseofthevariablewedeclaredbefore.Wereturnaprojectionwithanarrayoftheusernamesandthecalculatedage,forwhichweusethevariableagain:

[

{

"pair":["JohnSmith","KatieFoster"],

"sumOfAges":72

},

{

"pair":["KatieFoster","JohnSmith"],

"sumOfAges":72

}

]

Protip:whendefiningobjects,ifthedesiredattributekeyandthevariabletousefortheattributevaluearethesame,youcanuseashorthandnotation:{sumOfAges}insteadof{sumOfAges:sumOfAges}.

Finally,let'sdeleteoneoftheuserdocuments:

REMOVE"9883"INusers

ItdeletestheuserJohn(_key:"9883").Wecouldalsoremovedocumentsinaloop(samegoesforINSERT,UPDATEandREPLACE):

FORuserINusers

FILTERuser.age>=30

REMOVEuserINusers

GettingStarted

16

Thequerydeletesalluserswhoseageisgreaterthanorequalto30.

Howtocontinue

ThereisalotmoretodiscoverinAQLandmuchmorefunctionalitythatArangoDBoffers.Continuereadingtheotherchaptersandexperimentwithatestdatabasetofosteryourknowledge.

IfyouwanttowritemoreAQLqueriesrightnow,havealookhere:

DataQueries:dataaccessandmodificationqueriesHigh-leveloperations:detaileddescriptionsofFOR,FILTERandmoreoperationsnotshowninthisintroductionFunctions:areferenceofallprovidedfunctions

ArangoDBprogramsTheArangoDBpackagecomeswiththefollowingprograms:

arangod:TheArangoDBdatabasedaemon.ThisserverprogramisintendedtorunasadaemonprocessandtoservethevariousclientsconnectiontotheserverviaTCP/HTTP.

arangosh:TheArangoDBshell.Aclientthatimplementsaread-eval-printloop(REPL)andprovidesfunctionstoaccessandadministratetheArangoDBserver.

arangoimp:AbulkimporterfortheArangoDBserver.ItsupportsJSONandCSV.

arangodump:AtooltocreatebackupsofanArangoDBdatabaseinJSONformat.

arangorestore:AtooltoloaddataofabackupbackintoanArangoDBdatabase.

arango-dfdb:AdatafiledebuggerforArangoDB.ItisprimarilyintendedtobeusedduringdevelopmentofArangoDB.

arangobench:Abenchmarkandtesttool.Itcanbeusedforperformanceandserverfunctiontesting.

GettingStarted

17

InstallingFirstofall,downloadandinstallthecorrespondingRPMorDebianpackageorusehomebrewonMacOSX.Youcanfindpackagesforvariousoperationsystemsatourinstallsection,includinginstallersforWindows.

Howtodothatindetailisdescribedinthesubchaptersofthissection.

Onhowtosetupacluster,checkouttheDeploymentchapter.

Installing

18

https://www.arangodb.com/download

LinuxVisittheofficialArangoDBdownloadpageanddownloadthecorrectpackageforyourLinuxdistribution.Youcanfindbinarypackagesforthemostcommondistributionsthere.Followtheinstructionstouseyourfavoritepackagemanagerforthemajordistributions.AftersettinguptheArangoDBrepositoryyoucaneasilyinstallArangoDBusingyum,aptitude,urpmiorzypper.Debianbasedpackageswillaskforapasswordduringinstallation.ForanunattendedinstallationforDebian,seebelow.Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Forotherdistributionsortochangethepassword,runarango-secure-installationtosetarootpassword.Alternatively,seeCompilingifyouwanttobuildArangoDByourself.Startupthedatabaseserver.

Normally,thisisdonebyexecutingthefollowingcommand:

unix>/etc/init.d/arangodstart

Itwillstarttheserver,anddothataswellatsystemboottime.

Tostoptheserveryoucanusethefollowingcommand:

unix>/etc/init.d/arangodstop

TheexactcommandsdependonyourLinuxdistribution.Youmayrequirerootprivilegestoexecutethesecommands.

LinuxMint

PleaseusethecorrespondingUbuntuorDebianpackages.

UnattendedInstallation

Debianbasedpackagewillaskforapasswordduringinstallation.Forunattendedinstallation,youcansetthepasswordusingthedebconfhelpers.

echoarangodb3arangodb3/passwordpasswordNEWPASSWORD|debconf-set-selections

echoarangodb3arangodb3/password_againpasswordNEWPASSWORD|debconf-set-selections

Thecommandsshouldbeexecutedpriortotheinstallation.

Red-Hatbasedpackageswillsetarandompasswordduringinstallation.Ifyouwanttoforceapassword,execute

ARANGODB_DEFAULT_ROOT_PASSWORD=NEWPASSWORDarango-secure-installation

Thecommandshouldbeexecutedaftertheinstallation.

Non-StandardInstallationIfyoucompiledArangoDBfromsourceanddidnotuseanyinstallationpackage–orusingnon-defaultlocationsand/ormultipleArangoDBinstancesonthesamehost–youmaywanttostarttheserverprocessmanually.Youcandosobyinvokingthearangodbinaryfromthecommandlineasshownbelow:

unix>/usr/local/sbin/arangod/tmp/vocbase

20ZZ-XX-YYT12:37:08Z[8145]INFOusingbuilt-inJavaScriptstartupfiles

20ZZ-XX-YYT12:37:08Z[8145]INFOArangoDB(version1.x.y)isreadyforbusiness

20ZZ-XX-YYT12:37:08Z[8145]INFOHaveFun!

Linux

19


http://www.microhowto.info/howto/perform_an_unattended_installation_of_a_debian_package.html

Tostopthedatabaseservergracefully,youcaneitherpressCTRL-CorbysendtheSIGINTsignaltotheserverprocess.Onmanysystemsthiscanbeachievedwiththefollowingcommand:

unix>kill-2`pidofarangod`

Onceyoustartedtheserver,thereshouldbearunninginstanceofarangod-theArangoDBdatabaseserver.

unix>psauxw|fgreparangod

arangodb145360.10.6530726423464s002S1:21pm0:00.18/usr/local/sbin/arangod

Ifthereisnosuchprocess,checkthelogfile/var/log/arangodb/arangod.logforerrors.Ifyouseealogmessagelike

2012-12-03T11:35:29Z[12882]ERRORDatabasedirectoryversion(1)islowerthanserverversion(1.2).

2012-12-03T11:35:29Z[12882]ERRORItseemslikeyouhaveupgradedtheArangoDBbinary.Ifthisiswhatyouwantedtodo,pleas

erestartwiththe--database.auto-upgradeoptiontoupgradethedatainthedatabasedirectory.

2012-12-03T11:35:29Z[12882]FATALDatabaseversioncheckfailed.Pleasestarttheserverwiththe--database.auto-upgradeopti

on

makesuretostarttheserveroncewiththe--database.auto-upgradeoption.

Notethatyoumayhavetoenableloggingfirst.Ifyoustarttheserverinashell,youshouldseeerrorsloggedthereaswell.

Linux

20

MacOSXThepreferredmethodforinstallingArangoDBunderMacOSXishomebrew.However,incaseyouarenotusinghomebrew,weprovideacommand-lineapporgraphicalappwhichcontainsalltheexecutables.

Homebrew

Ifyouareusinghomebrew,thenyoucaninstallthelatestreleasedstableversionofArangoDBusingbrewasfollows:

brewinstallarangodb

ThiswillinstallthecurrentstableversionofArangoDBandalldependencieswithinyourHomebrewtree.Notethattheserverwillbeinstalledas:

/usr/local/sbin/arangod

Youcanstarttheserverbyrunningthecommand/usr/local/sbin/arangod&.

Configurationfileislocatedat

/usr/local/etc/arangodb3/arangod.conf

TheArangoDBshellwillbeinstalledas:

/usr/local/bin/arangosh

YoucanuninstallArangoDBusing:

brewuninstallarangodb

However,incaseyoustartedArangoDBusingthelaunchctl,youneedtounloaditbeforeuninstallingtheserver:

launchctlunload~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

ThenremovetheLaunchAgent:

rm~/Library/LaunchAgents/homebrew.mxcl.arangodb.plist

Note:IfthelatestArangoDBVersionisnotshowninhomebrew,youalsoneedtoupdatehomebrew:

brewupdate

Knownissues

Performance-theLLVMdeliveredasofMacOSXElCapitanbuildsslowbinaries.UseGCCinstead,untilthisissuehasbeenfixedbyApple.theCommandlineargumentparsingdoesn'tacceptblanksinfilenames;theCLIversionbelowdoes.ifyouneedtochangeserverendpointwhilestartinghomebrewversion,youcaneditarangod.conffileanduncommentlinewithendpointneeded,e.g.:

[server]

endpoint=tcp://0.0.0.0:8529

MacOSX

21

http://brew.sh/

GraphicalApp

Incaseyouarenotusinghomebrew,wealsoprovideagraphicalapp.Youcandownloaditfromhere.

ChooseMacOSX.DownloadandinstalltheapplicationArangoDBinyourapplicationfolder.

CommandlineAppIncaseyouarenotusinghomebrew,wealsoprovideacommand-lineapp.Youcandownloaditfromhere.

ChooseMacOSX.DownloadandinstalltheapplicationArangoDB-CLIinyourapplicationfolder.

Startingtheapplicationwillstarttheserverandopenaterminalwindowshowingyouthelog-file.

ArangoDBserverhasbeenstarted

Thedatabasedirectoryislocatedat

'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/lib/arangodb'

Thelogfileislocatedat

'/Applications/ArangoDB-CLI.app/Contents/MacOS/opt/arangodb/var/log/arangodb/arangod.log'

Youcanaccesstheserverusingabrowserat'http://127.0.0.1:8529/'

orstarttheArangoDBshell

'/Applications/ArangoDB-CLI.app/Contents/MacOS/arangosh'

Switchingtolog-filenow,killingthiswindowswillNOTstoptheserver.

2013-10-27T19:42:04Z[23840]INFOArangoDB(version1.4.devel[darwin])isreadyforbusiness.Havefun!

Notethatitispossibletoinstallboth,thehomebrewversionandthecommand-lineapp.Youshould,however,edittheconfigurationfilesofoneversionandchangetheportused.

MacOSX

22



WindowsThedefaultinstallationdirectoryisC:\ProgramFiles\ArangoDB-3.x.x.Duringtheinstallationprocessyoumaychangethis.InthefollowingdescriptionwewillassumethatArangoDBhasbeeninstalledinthelocation<ROOTDIR>.

Youhavetobecarefulwhenchoosinganinstallationdirectory.Youneedeitherwritepermissiontothisdirectoryoryouneedtomodifytheconfigurationfilefortheserverprocess.InthelattercasethedatabasedirectoryandtheFoxxdirectoryhavetobewritablebytheuser.

Single-andMultiuserInstallation

TherearetwomainmodesfortheinstallerofArangoDB.Theinstallerletsyouselect:

multiuserinstallation(default;adminprivilegesrequired)WillinstallArangoDBasservice.singleuserinstallationAllowtoinstallArangodbasnormaluser.Requiresmanualstartingofthedatabaseserver.

CheckBoxesThecheckboxesallowyoutochoseweatheryouwantto:

chosecustominstallpathsdoanautomaticupgradekeepanbackupofyourdataaddexecutablestopathcreateadesktopicon

ornot.

CustomInstallPaths

Thischeckboxcontrolsifyouwillbeabletooverridethedefaultpathsfortheinstallationinsubsequentsteps.

Thedefaultinstallationpathsare:

MultiUserDefault:

Installation:C:\ProgramFiles\ArangoDB-3.x.xDataBase:C:\ProgramData\ArangoDBFoxxService:C:\ProgramData\ArangoDB-apps

SingleUserDefault:

Installation:C:\Users\\\AppData\Local\ArangoDB-3.x.xDataBase:C:\Users\\\AppData\Local\ArangoDBFoxxService:C:\Users\\\AppData\Local\ArangoDB-apps

Wearenotusingtheroamingpartoftheuser'sprofile,becausedoingsoavoidsthedatabeingsyncedtothewindowsdomaincontroller.

AutomaticUpgrade

Ifthischeckboxisselectedtheinstallerwillattempttoperformanautomaticupdate.FormoreinformationpleaseseeUpgradingfromPreviousVersion.

KeepBackup

Selectthistocreateabackupofyourdatabasedirectoryduringautomaticupgrade.Thebackupwillbecreatednexttoyourcurrentdatabasedirectorysuffixedbyatimestamp.

Windows

23

AddtoPath

Selectthistoaddthebinarydirectorytoyoursystem'spath(multiuserinstallation)oruser'spath(singleuserinstallation).

DesktopIcon

SelectifyouwanttheinstallertocreateDesktopIconsthatletyou:

accessthewebintefacestartthecommandlineclient(arangosh)startthedatabaseserver(singleuserinstallationonly)

UpgradingfromPreviousVersionIfyouareupgradingArangoDBfromanearlierversionyouneedtocopyyourolddatabasedirectorytothenewdefaultpaths.Upgradingwillkeepyourolddata,passwordandchoiceofstorageengineasitis.SwitchingtotheRocksDBstorageenginerequiresaexportandreimportofyourdata.

StartingIfyouinstalledArangoDBformultipleusers(asaservice)itisautomaticallystarted.OtherwiseyouneedtousethelinkthatwascreatedonyouDesktopifyouchosetolettheinstallercreatedesktopiconsor

theexecutablearangod.exelocatedin<ROOTDIR>\bin.Thiswillusetheconfigurationfilearangod.conflocatedin<ROOTDIR>\etc\arangodb,whichyoucanadjusttoyourneedsandusethedatadirectory<ROOTDIR>\var\lib\arangodb.Thisistheplacewhereallyourdata(databasesandcollections)willbestoredbydefault.

Pleasechecktheoutputofthearangod.exeexecutablebeforegoingon.Iftheserverstartedsuccessfully,youshouldseealineArangoDBisreadyforbusiness.Havefun!attheendofitsoutput.

Wenowwishtocheckthattheinstallationisworkingcorrectlyandtodothiswewillbeusingtheadministrationwebinterface.Executearangod.exeifyouhavenotalreadydoneso,thenopenupyourwebbrowserandpointittothepage:

http://127.0.0.1:8529/

AdvancedStarting

Ifyouwanttoprovideourownstartscripts,youcansettheenvironmentvariableARANGODB_CONFIG_PATH.Thisvariableshouldpointtoadirectorycontainingtheconfigurationfiles.

UsingtheClient

ToconnecttoanalreadyrunningArangoDBserverinstance,thereisashellarangosh.exelocatedin<ROOTDIR>\bin.Thisstartsashellwhichcanbeused–amongstotherthings–toadministerandqueryalocalorremoteArangoDBserver.

Notethatarangosh.exedoesNOTstartaseparateserver,itonlystartstheshell.Touseityoumusthaveaserverrunningsomewhere,e.g.byusingthearangod.exeexecutable.

arangosh.exeusesconfigurationfromthefilearangosh.conflocatedin<ROOTDIR>\etc\arangodb\.Pleaseadjustthistoyourneedsifyouwanttousedifferentconnectionsettingsetc.

Uninstalling

TouninstalltheArangoserverapplicationyoucanusethewindowscontrolpanel(asyouwouldnormallyuninstallanapplication).Notehowever,thatanydatafilescreatedbytheArangoserverwillremainaswellasthe<ROOTDIR>directory.Tocompletetheuninstallationprocess,removethedatafilesandthe<ROOTDIR>directorymanually.

Windows

24

LimitationsforCygwin

PleasenotesomeimportantlimitationswhenrunningArangoDBunderCygwin:StartingArangoDBcanbestartedfromoutofaCygwinterminal,butpressingCTRL-Cwillforcefullykilltheserverprocesswithoutgivingitachancetohandlethekillsignal.Inthiscase,aregularservershutdownisnotpossible,whichmayleaveafileLOCKaroundintheserver'sdatadirectory.ThisfileneedstoberemovedmanuallytomakeArangoDBstartagain.Additionally,asArangoDBdoesnothaveachancetohandlethekillsignal,theservercannotforcefullyflushanydatatodiskonshutdown,leadingtopotentialdataloss.WhenstartingArangoDBfromaCygwinterminalitmightalsohappenthatnoerrorsareprintedintheterminaloutput.StartingArangoDBfromanMS-DOScommandpromptdoesnotimposetheselimitationsandisthusthepreferredmethod.

PleasenotethatArangoDBusesUTF-8asitsinternalencodingandthatthesystemconsolemustsupportaUTF-8codepage(65001)andfont.ItmaybenecessarytomanuallyswitchtheconsolefonttoafontthatsupportsUTF-8.

Windows

25

CompilingArangoDBfromscratchThefollowingsectionsdescribehowtocompileandbuildtheArangoDBfromscratch.ArangoDBwillcompileonmostLinuxandMacOSXsystems.WeassumethatyouusetheGNUC/C++compilerorclang/clang++tocompilethesource.ArangoDBhasbeentestedwiththesecompilers,butshouldbeabletocompilewithanyPosix-compliant,C++11-enabledcompiler.PleaseletusknowwhetheryousuccessfullycompileditwithanotherC/C++compiler.

Bydefault,cloningthegithubrepositorywillcheckoutdevel.ThisversioncontainsthedevelopmentversionoftheArangoDB.UsethisbranchifyouwanttomakechangestotheArangoDBsource.

OnWindowsyoufirstneedtoallowandenablesymlinksforyouruser.

PleasecheckoutthecookbookonhowtocompileArangoDB.

Compiling

26

https://github.com/git-for-windows/git/wiki/Symbolic-Links#allowing-non-administrators-to-create-symbolic-links

AuthenticationArangoDBallowstorestrictaccesstodatabasestocertainusers.Allusersofthesystemdatabaseareconsideredadministrators.Duringinstallationadefaultuserrootiscreated,whichhasaccesstoalldatabases.

Youshouldcreateadatabaseforyourapplicationtogetherwithauserthathasaccessrightstothisdatabase.SeeManagingUsers.

Usethearangoshtocreateanewdatabaseanduser.

arangosh>db._createDatabase("example");

arangosh>varusers=require("@arangodb/users");

arangosh>users.save("root@example","password");

arangosh>users.grantDatabase("root@example","example");

Youcannowconnecttothenewdatabaseusingtheuserroot@example.

shell>arangosh--server.username"root@example"--server.databaseexample

Authentication

27

AccessingtheWebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.ThewebinterfacecanbeaccessedviatheURL:

http://127.0.0.1:8529

Ifeverythingworksasexpected,youshouldseetheloginview:

FormoreinformationontheArangoDBwebinterface,seeWebInterface

AccessingtheWebInterface

28

ComingfromSQLIfyouworkedwitharelationaldatabasemanagementsystem(RDBMS)suchasMySQL,MariaDBorPostgreSQL,youwillbefamiliarwithitsquerylanguage,adialectofSQL(StructuredQueryLanguage).

ArangoDB'squerylanguageiscalledAQL.Therearesomesimilaritiesbetweenbothlanguagesdespitethedifferentdatamodelsofthedatabasesystems.ThemostnotabledifferenceisprobablytheconceptofloopsinAQL,whichmakesitfeelmorelikeaprogramminglanguage.Itsuitstheschema-lessmodelmorenaturalandmakesthequerylanguageverypowerfulwhileremainingeasytoreadandwrite.

TogetstartedwithAQL,havealookatourdetailedcomparisonofSQLandAQL.ItwillalsohelpyoutotranslateSQLqueriestoAQLwhenmigratingtoArangoDB.

Howdobrowsevectorstranslateintodocumentqueries?

IntraditionalSQLyoumayeitherfetchallcolumnsofatablerowbyrow,usingSELECT*FROMtable,orselectasubsetofthecolumns.Thelistoftablecolumnstofetchiscommonlycalledcolumnlistorbrowsevector:

SELECTcolumnA,columnB,columnZFROMtable

Sincedocumentsaren'ttwo-dimensional,andneitherdoyouwanttobelimitedtoreturningtwo-dimensionallists,therequirementsforaquerylanguagearehigher.AQListhusalittlebitmorecomplexthanplainSQLatfirst,butoffersmuchmoreflexibilityinthelongrun.Itletsyouhandlearbitrarilystructureddocumentsinconvenientways,mostlyleanedonthesyntaxusedinJavaScript.

Composingthedocumentstobereturned

TheAQLRETURNstatementreturnsoneitemperdocumentitishanded.Youcanreturnthewholedocument,orjustpartsofit.GiventhatoneDocumentisadocument(retrievedlikeLEToneDocument=DOCUMENT("myusers/3456789")forinstance),itcanbereturnedas-islikethis:

RETURNoneDocument

[

{

"_id":"myusers/3456789",

"_key":"3456789"

"_rev":"14253647",

"firstName":"John",

"lastName":"Doe",

"address":{

"city":"Gotham",

"street":"RoadToNowhere1"

},

"hobbies":[

{name:"swimming",howFavorite:10},

{name:"biking",howFavorite:6},

{name:"programming",howFavorite:4}

]

}

]

Returnthehobbiessub-structureonly:

RETURNoneDocument.hobbies

[

[




]

ComingfromSQL

29

https://arangodb.com/why-arangodb/sql-aql-comparison/

]

Returnthehobbiesandtheaddress:

RETURN{

hobbies:oneDocument.hobbies,

address:oneDocument.address

}

[

{

hobbies:[




],

address:{

"city":"Gotham",

"street":"RoadToNowhere1"

}

}

]

Returnthefirsthobbyonly:

RETURNoneDocument.hobbies[0].name

[

"swimming"

]

Returnalistofallhobbystrings:

RETURN{hobbies:oneDocument.hobbies[*].name}

[

{hobbies:["swimming","biking","porgramming"]}

]

MorecomplexarrayandobjectmanipulationscanbedoneusingAQLfunctionsandoperators.

ComingfromSQL

30

TutorialsKubernetes:StartArangoDBonKubernetesin5minutes

Kubernetes:DC2DConKubernetes

Tutorials

31

StartArangoDBonKubernetesin5minutesStartinganArangoDBdatabase(eithersingleserverorfullblowncluster)onKubernetesinvolvesalotofresources.

TheserversneedstoruninPods,youneedSecretsforauthentication,TLScertificatesandServicestoenablecommunicationwiththedatabase.

Usekube-arangodb,theArangoDBKubernetesOperatortogreatlysimplifythisprocess.

Inthisguide,wewillexplainwhattheArangoDBKubernetesOperatoris,howtoinstallitandhowuseittodeployyourfirstArangoDBdatabaseinaKubernetescluster.

Whatis kube-arangodb

kube-arangodbisasetoftwooperatorsthatyoudeployinyourKubernetesclusterto(1)managedeploymentsoftheArangoDBdatabaseand(2)providePersistentVolumesonlocalstorageofyournodesforoptimalstorageperformance.

NotethattheoperatorthatprovidesPersistentVolumesisnotneededtorunArangoDBdeployments.YoucanalsousePersistentVolumesprovidedbyothercontrollers.

InthisguidewewillfocusontheArangoDeploymentoperator.

Installing kube-arangodb

Toinstallkube-arangodbinyourKubernetescluster,makesureyouhaveaccesstothisclusterandtherightstodeployresourcesatclusterlevel.

Fornow,anyrecentKubernetesclusterwilldo(e.g.minikube).

Thenrun(replace<version>withtheversionoftheoperatorthatyouwanttoinstall):

kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests/crd.yaml

kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests/arango-deployment.yaml

#Optional

kubectlapply-fhttps://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests/arango-storage.yaml

ThefirstcommandinstallstwoCustomResourceDefinitionsinyourKubernetescluster:

ArangoDeploymentistheresourceusedtodeployArangoDBdatabase.ArangoLocalStorageistheresourceusedtoprovisionPersistentVolumesonlocalstorage.

ThesecondcommandinstallsaDeploymentthatrunstheoperatorthatcontrolsArangoDeploymentresources.

TheoptionalthirdcommandinstallsaDeploymentthatrunstheoperatorthatprovidesPersistentVolumesonlocaldisksoftheclusternodes.Usethiswhenrunningonbare-metalorifthereisnoprovisionerforfaststorageinyourKubernetescluster.

DeployingyourfirstArangoDBdatabaseThefirstdatabasewearegoingtodeployisasingleserverdatabase.

Createafilecalledsingle-server.yamlwiththefollowingcontent.

apiVersion:"database.arangodb.com/v1alpha"

kind:"ArangoDeployment"

metadata:

name:"single-server"

spec:

mode:Single

Kubernetes

32

NowinsertthisresourceinyourKubernetesclusterusing:

kubectlapply-fsingle-server.yaml

TheArangoDeploymentoperatorinkube-arangodbwillnowinspecttheresourceyoujustdeployedandstarttheprocesstorunasingleserverdatabase.

Toinspectthecurrentstatusofyourdeployment,run:

kubectldescribeArangoDeploymentsingle-server

#orshorter

kubectldescribearangosingle-server

Toinspectthepodscreatedforthisdeployment,run:

kubectlgetpods--selector=arango_deployment=single-server

Theresultwilllooksimilartothis:

NAMEREADYSTATUSRESTARTSAGE

single-server-sngl-cjtdxrgl-fe06f01/1Running01m

OncethepodreportsthatitishasaRunningstatusandisready,yourdatabasesavailable.

Connectingtoyourdatabase

ThesingleserverdatabaseyoudeployedinthepreviouschapterisnowavailablefromwithintheKubernetesclusteraswellasoutsideit.

AccesstothedatabasefromoutsidetheKubernetesclusterisprovidedusinganexternal-accessservice.BydefaultthisserviceisoftypeLoadBalancer.IfthistypeofserviceisnotsupportedbyyourKubernetescluster,itwillbereplacedbyaserviceoftypeNodePortafteraminute.

Toseethetypeofservicethathasbeencreated,run:

kubectlgetservicesingle-server-ea

WhentheserviceisoftheLoadBalancertype,usetheIPaddresslistedintheEXTERNAL-IPcolumnwithport8529.WhentheserviceisoftheNodePorttype,usetheIPaddressofanyofthenodesofthecluster,combinewiththehigh(>30000)portlistedinthePORT(S)column.

Nowyoucanconnectyourbrowsertohttps://<ip>:<port>/.

Yourbrowserwillshowawarningaboutanunknowncertificate.Acceptthecertificatefornow.

Thenloginusingusernamerootandanemptypassword.

IfyouwanttodeleteyoursingleserverArangoDBdatabase,justrun:

kubectldeleteArangoDeploymentsingle-server

DeployingafullblownArangoDBclusterdatabase

Thedeploymentofafullblownclusterisverysimilartodeployingasingleserverdatabase.ThedifferenceisinthemodefieldoftheArangoDeploymentspecification.

Createafilecalledcluster.yamlwiththefollowingcontent.



Kubernetes

33

metadata:

name:"cluster"

spec:

mode:Cluster

NowinsertthisresourceinyourKubernetesclusterusing:

kubectlapply-fcluster.yaml

Thesamecommandsusedinthesingleserverdeploymentcanbeusedtoinspectyourcluster.Justusethecorrectdeploymentname(clusterinsteadofsingle-server).

Wheretogofromhere

ArangoDBKubernetesOperator

Kubernetes

34

StartArangoDBClustertoClusterSynchronizationonKubernetesThistutorialguidesyouthroughthestepsneededtoconfigureanArangoDBdatacentertodatacenterreplicationbetweentwoArangoDBclustersrunninginKubernetes.

Requirements

1. Thistutorialassumesthatyouhave2ArangoDBclustersrunningin2differentKubernetesclusters.2. BothKubernetesclustersareequippedwithsupportforServicesoftypeLoadBalancer.3. Youcancreate(global)DNSnamesforconfiguredServiceswithlowpropagationtimes.E.g.useCloudflare.4. Youhave4DNSnamesavailable:

OneforthedatabaseinthesourceArangoDBcluster.E.g.src-db.mycompany.comOnefortheArangoDBsyncmastersinthesourceArangoDBcluster.E.g.src-sync.mycompany.comOneforthedatabaseinthedestinationArangoDBcluster.E.g.dst-db.mycompany.comOnefortheArangoDBsyncmastersinthedestinationArangoDBcluster.E.g.dst-sync.mycompany.com

Step1:EnableDatacenterReplicationSupportonsourceArangoDBcluster

SetyourcurrentKubernetescontexttotheKubernetessourcecluster.

EdittheArangoDeploymentofthesourceArangoDBclusters.

Set:

spec.tls.altNamesto["src-db.mycompany.com"](canincludemorenames/IPaddresses)spec.sync.enabledtotruespec.sync.externalAccess.masterEndpointto["https://src-sync.mycompany.com:8629"]spec.sync.externalAccess.accessPackageSecretNamesto["src-accesspackage"]

Step2:Extractaccess-packagefromsourceArangoDBcluster

Run:

kubectlgetsecretsrc-accesspackage--template='{{index.data"accessPackage.yaml"}}'|\

base64-D>accessPackage.yaml

Step3:ConfiguresourceDNSnames

Run:

kubectlgetservice

FindtheIPaddresscontainedintheLoadBalancercolumnforthefollowingServices:

<deployment-name>-eaUsethisIPaddressforthesrc-db.mycompany.comDNSname.<deployment-name>-syncUsethisIPaddressforthesrc-sync.mycompany.comDNSname.

TheprocessforconfiguringDNSnamesisspecifictoeachDNSprovider.


35

Step4:EnableDatacenterReplicationSupportondestinationArangoDBcluster

SetyourcurrentKubernetescontexttotheKubernetesdestinationcluster.

EdittheArangoDeploymentofthesourceArangoDBclusters.

Set:

spec.tls.altNamesto["dst-db.mycompany.com"](canincludemorenames/IPaddresses)spec.sync.enabledtotruespec.sync.externalAccess.masterEndpointto["https://dst-sync.mycompany.com:8629"]

Step5:ImportaccesspackageindestinationclusterRun:

kubectlapply-faccessPackage.yaml

Note:ThisimportstwoSecrets,containingTLSinformationaboutthesourcecluster,intothedestinationcluster

Step6:ConfiguredestinationDNSnames

Run:

kubectlgetservice

FindtheIPaddresscontainedintheLoadBalancercolumnforthefollowingServices:

<deployment-name>-eaUsethisIPaddressforthedst-db.mycompany.comDNSname.<deployment-name>-syncUsethisIPaddressforthedst-sync.mycompany.comDNSname.

TheprocessforconfiguringDNSnamesisspecifictoeachDNSprovider.

Step7:Createan ArangoDeploymentReplicationresourceCreateayamlfile(e.g.calledsrc-to-dst-repl.yaml)withthefollowingcontent:

apiVersion:"replication.database.arangodb.com/v1alpha"

kind:"ArangoDeploymentReplication"

metadata:

name:"replication-src-to-dst"

spec:

source:

masterEndpoint:["https://src-sync.mycompany.com:8629"]

auth:

keyfileSecretName:src-accesspackage-auth

tls:

caSecretName:src-accesspackage-ca

destination:

deploymentName:<dst-deployment-name>

Step8:WaitforDNSnamestopropagateWaituntiltheDNSnamesconfiguredinstep3and6resolvetotheirconfiguredIPaddresses.

DependingonyourDNSprovidesthiscantakeafewminutesupto24hours.


36

Step9:Activatereplication

Run:

kubectlapply-fsrc-to-dst-repl.yaml

Replicationfromthesourceclustertothedestinationclusterwillnowbeconfigured.

CheckthestatusofthereplicationbyinspectingthestatusoftheArangoDeploymentReplicationresourceusing:

kubectldescribeArangoDeploymentReplicationreplication-src-to-dst

Assoonasthereplicationisconfigured,theAddcollectionbuttonintheCollectionspageofthewebUI(ofthedestinationcluster)willbegrayedout.


37

Highlights

Version3.3

EnterpriseEdition

DatacentertoDatacenterReplication:ReplicatetheentirestructureandcontentofanArangoDBclusterasynchronouslytoanotherclusterinadifferentdatacenterwithArangoSync.Multi-datacentersupportmeansyoucanfallbacktoareplicaofyourclusterincaseofadisasterinonedatacenter.

EncryptedBackups:ArangodumpcancreatebackupsencryptedwithasecretkeyusingAES256blockcipher.

AllEditions

Server-levelReplication:Inadditiontoper-databasereplication,thereisnowanadditionalglobalApplier.Starttheglobalreplicationontheslaveonceandallcurrentandfuturedatabaseswillbereplicatedfromthemastertotheslaveautomatically.

AsynchronousFailover:Makeasingleserverinstanceresilientwithasecondserverinstance,oneasmasterandtheotherasasynchronouslyreplicatingslave,withautomaticfailovertotheslaveifthemastergoesdown.

AlsoseeWhat'sNewin3.3.

Version3.2

RocksDBStorageEngine:YoucannowuseasmuchdatainArangoDBasyoucanfitonyourdisk.Plus,youcanenjoyperformanceboostsonwritesbyhavingonlydocument-levellocks

Pregel:WeimplementeddistributedgraphprocessingwithPregeltodiscoverhiddenpatterns,identifycommunitiesandperformin-depthanalyticsoflargegraphdatasets.

Fault-TolerantFoxx:TheFoxxmanagementinternalshavebeenrewrittenfromthegrounduptomakesuremulti-coordinatorclustersetupsalwayskeeptheirservicesinsyncandnewcoordinatorsarefullyinitializedevenwhenallexistingcoordinatorsareunavailable.

Enterprise:Workingwithsomeofourlargestcustomers,we’veaddedfurthersecurityandscalabilityfeaturestoArangoDBEnterpriselikeLDAPintegration,EncryptionatRest,andthebrandnewSatelliteCollections.


Version3.1SmartGraphs:Scalewithgraphstoaclusterandstayperformant.WithSmartGraphsyoucanusethe"smartness"ofyourapplicationlayertoshardyourgraphefficientlytoyourmachinesandlettraversalsrunlocally.

EncryptionControl:ChooseyourlevelofSSLencryption

Auditing:KeepadetailedlogofalltheimportantthingsthathappenedinArangoDB.


Version3.0

self-organizingclusterwithsynchronousreplication,master/mastersetup,sharednothingarchitecture,clustermanagementagency.

Deeplyintegrated,nativeAQLgraphtraversal

Highlights

38

VelocyPackasnewinternalbinarystorageformataswellasforintermediateAQLvalues.

PersistentindexesviaRocksDBsuitableforsortingandrangequeries.

Foxx3.0:overhauledJSframeworkfordata-centricmicroservices

SignificantlyimprovedWebInterface


Highlights

39

https://github.com/arangodb/velocypack

ScalabilityArangoDBisadistributeddatabasesupportingmultipledatamodels,andcanthusbescaledhorizontally,thatis,byusingmanyservers,typicallybasedoncommodityhardware.Thisapproachnotonlydeliversperformanceaswellascapacityimprovements,butalsoachievesresiliencebymeansofreplicationandautomaticfail-over.Furthermore,onecanbuildsystemsthatscaletheircapacitydynamicallyupanddownautomaticallyaccordingtodemand.

OnecanalsoscaleArangoDBvertically,thatis,byusingeverlargerservers.ThereisnobuiltinlimitationinArangoDB,forexample,theserverwillautomaticallyusemorethreadsifmoreCPUsarepresent.

However,scalingverticallyhasthedisadvantagethatthecostsgrowfasterthanlinearwiththesizeoftheserver,andnoneoftheresilienceanddynamicalcapabilitiescanbeachievedinthisway.

InthischapterweexplainthedistributedarchitectureofArangoDBanddiscussitsscalabilityfeaturesandlimitations:

ArangoDB'sdistributedarchitectureDifferentdatamodelsandscalabilityLimitations

Scalability

40

Architecture

TheclusterarchitectureofArangoDBisaCPmaster/mastermodelwithnosinglepointoffailure.With"CP"wemeanthatinthepresenceofanetworkpartition,thedatabaseprefersinternalconsistencyoveravailability.With"master/master"wemeanthatclientscansendtheirrequeststoanarbitrarynode,andexperiencethesameviewonthedatabaseregardless."Nosinglepointoffailure"meansthattheclustercancontinuetoserverequests,evenifonemachinefailscompletely.

Inthisway,ArangoDBhasbeendesignedasadistributedmulti-modeldatabase.Thissectiongivesashortoutlineontheclusterarchitectureandhowtheabovefeaturesandcapabilitiesareachieved.

StructureofanArangoDBcluster

AnArangoDBclusterconsistsofanumberofArangoDBinstanceswhichtalktoeachotheroverthenetwork.Theyplaydifferentroles,whichwillbeexplainedindetailbelow.Thecurrentconfigurationoftheclusterisheldinthe"Agency",whichisahighly-availableresilientkey/valuestorebasedonanoddnumberofArangoDBinstancesrunningRaftConsensusProtocol.

ForthevariousinstancesinanArangoDBclusterthereare4distinctroles:Agents,Coordinators,PrimaryandSecondaryDBservers.Inthefollowingsectionswewillshedlightoneachofthem.NotethatthetasksforallrolesrunthesamebinaryfromthesameDockerimage.

Agents

OneormultipleAgentsformtheAgencyinanArangoDBcluster.TheAgencyisthecentralplacetostoretheconfigurationinacluster.Itperformsleaderelectionsandprovidesothersynchronizationservicesforthewholecluster.WithouttheAgencynoneoftheothercomponentscanoperate.

Whilegenerallyinvisibletotheoutsideitistheheartofthecluster.Assuch,faulttoleranceisofcourseamusthavefortheAgency.ToachievethattheAgentsareusingtheRaftConsensusAlgorithm.ThealgorithmformallyguaranteesconflictfreeconfigurationmanagementwithintheArangoDBcluster.

AtitscoretheAgencymanagesabigconfigurationtree.Itsupportstransactionalreadandwriteoperationsonthistree,andotherserverscansubscribetoHTTPcallbacksforallchangestothetree.

Coordinators

Coordinatorsshouldbeaccessiblefromtheoutside.Thesearetheonestheclientstalkto.TheywillcoordinateclustertaskslikeexecutingqueriesandrunningFoxxservices.Theyknowwherethedataisstoredandwilloptimizewheretorunusersuppliedqueriesorpartsthereof.Coordinatorsarestatelessandcanthuseasilybeshutdownandrestartedasneeded.

PrimaryDBservers

PrimaryDBserversaretheoneswherethedataisactuallyhosted.Theyhostshardsofdataandusingsynchronousreplicationaprimarymayeitherbeleaderorfollowerforashard.

Theyshouldnotbeaccessedfromtheoutsidebutindirectlythroughthecoordinators.Theymayalsoexecutequeriesinpartorasawholewhenaskedbyacoordinator.

Secondaries

SecondaryDBserversareasynchronousreplicasofprimaries.Ifoneisusingonlysynchronousreplication,onedoesnotneedsecondariesatall.Foreachprimary,therecanbeoneormoresecondaries.Sincethereplicationworksasynchronously(eventualconsistency),thereplicationdoesnotimpedetheperformanceoftheprimaries.Ontheotherhand,theirreplicaofthedatacanbeslightlyoutofdate.Thesecondariesareperfectlysuitableforbackupsastheydon'tinterferewiththenormalclusteroperation.

ClusterID

Everynon-AgencyArangoDBinstanceinaclusterisassignedauniqueIDduringitsstartup.UsingitsIDanodeisidentifiablethroughoutthecluster.AllclusteroperationswillcommunicateviathisID.

Architecture

41

https://raft.github.io/

https://raft.github.io/

Sharding

UsingtherolesoutlinedaboveanArangoDBclusterisabletodistributedatainsocalledshardsacrossmultipleprimaries.Fromtheoutsidethisprocessisfullytransparentandassuchweachievethegoalsofwhatothersystemscall"master-masterreplication".InanArangoDBclusteryoutalktoanycoordinatorandwheneveryoureadorwritedataitwillautomaticallyfigureoutwherethedataisstored(read)ortobestored(write).TheinformationabouttheshardsissharedacrossthecoordinatorsusingtheAgency.

AlsoseeShardingintheAdministrationchapter.

Manysensibleconfigurations

Thisarchitectureisveryflexibleandthusallowsmanyconfigurations,whicharesuitablefordifferentusagescenarios:

1. ThedefaultconfigurationistorunexactlyonecoordinatorandoneprimaryDBserveroneachmachine.Thisachievestheclassicalmaster/mastersetup,sincethereisaperfectsymmetrybetweenthedifferentnodes,clientscanequallywelltalktoanyoneofthecoordinatorsandallexposethesameviewtothedatastore.

2. OnecandeploymorecoordinatorsthanDBservers.ThisisasensibleapproachifoneneedsalotofCPUpowerfortheFoxxservices,becausetheyrunonthecoordinators.

3. OnecandeploymoreDBserversthancoordinatorsifmoredatacapacityisneededandthequeryperformanceisthelesserbottleneck

4. Onecandeployacoordinatoroneachmachinewhereanapplicationserver(e.g.anode.jsserver)runs,andtheAgentsandDBserversonaseparatesetofmachineselsewhere.Thisavoidsanetworkhopbetweentheapplicationserverandthedatabaseandthusdecreaseslatency.Essentially,thismovessomeofthedatabasedistributionlogictothemachinewheretheclientruns.

Theseshallsufficefornow.TheimportantpieceofinformationhereisthatthecoordinatorlayercanbescaledanddeployedindependentlyfromtheDBserverlayer.

Replication

ArangoDBofferstwowaysofdatareplicationwithinacluster,synchronousandasynchronous.Inthissectionweexplainsomedetailsandhighlighttheadvantagesanddisadvantagesrespectively.

Synchronousreplicationwithautomaticfail-over

Synchronousreplicationworksonaper-shardbasis.Oneconfiguresforeachcollection,howmanycopiesofeachshardarekeptinthecluster.Atanygiventime,oneofthecopiesisdeclaredtobethe"leader"andallotherreplicasare"followers".WriteoperationsforthisshardarealwayssenttotheDBserverwhichhappenstoholdtheleadercopy,whichinturnreplicatesthechangestoallfollowersbeforetheoperationisconsideredtobedoneandreportedbacktothecoordinator.Readoperationsareallservedbytheserverholdingtheleadercopy,thisallowstoprovidesnapshotsemanticsforcomplextransactions.

IfaDBserverfailsthatholdsafollowercopyofashard,thentheleadercannolongersynchronizeitschangestothatfollower.Afterashorttimeout(3seconds),theleadergivesuponthefollower,declaresittobeoutofsync,andcontinuesservicewithoutthefollower.Whentheserverwiththefollowercopycomesback,itautomaticallyresynchronizesitsdatawiththeleaderandsynchronousreplicationisrestored.

IfaDBserverfailsthatholdsaleadercopyofashard,thentheleadercannolongerserveanyrequests.ItwillnolongersendaheartbeattotheAgency.Therefore,asupervisionprocessrunningintheRaftleaderoftheAgency,cantakethenecessaryaction(after15secondsofmissingheartbeats),namelytopromoteoneoftheserversthatholdin-syncreplicasoftheshardtoleaderforthatshard.ThisinvolvesareconfigurationintheAgencyandleadstothefactthatcoordinatorsnowcontactadifferentDBserverforrequeststothisshard.Serviceresumes.Theothersurvivingreplicasautomaticallyresynchronizetheirdatawiththenewleader.WhentheDBserverwiththeoriginalleadercopycomesback,itnoticesthatitnowholdsafollowerreplica,resynchronizesitsdatawiththenewleaderandorderisrestored.

Allsharddatasynchronizationsaredoneinanincrementalway,suchthatresynchronizationsarequick.Thistechnologyallowstomoveshards(followerandleaderones)betweenDBserverswithoutserviceinterruptions.Therefore,anArangoDBclustercanmoveallthedataonaspecificDBservertootherDBserversandthenshutdownthatserverinacontrolledway.ThisallowstoscaledownanArangoDBclusterwithoutserviceinterruption,lossoffaulttoleranceordataloss.Furthermore,onecanre-balancethedistributionoftheshards,eithermanuallyorautomatically.

AlltheseoperationscanbetriggeredviaaREST/JSONAPIorviathegraphicalwebUI.Allfail-overoperationsarecompletelyhandledwithintheArangoDBcluster.

Architecture

42

Obviously,synchronousreplicationinvolvesacertainincreasedlatencyforwriteoperations,simplybecausethereisonemorenetworkhopwithintheclusterforeveryrequest.Thereforetheusercansetthereplicationfactorto1,whichmeansthatonlyonecopyofeachshardiskept,therebyswitchingoffsynchronousreplication.Thisisasuitablesettingforlessimportantoreasilyrecoverabledataforwhichlowlatencywriteoperationsmatter.

Asynchronousreplicationwithautomaticfail-over

Asynchronousreplicationworksdifferently,inthatitisorganizedusingprimaryandsecondaryDBservers.Eachsecondaryserverreplicatesallthedataheldonaprimarybypollinginanasynchronousway.Thisprocesshasverylittleimpactontheperformanceoftheprimary.Thedisadvantageisthatthereisadelaybetweentheconfirmationofawriteoperationthatissenttotheclientandtheactualreplicationofthedata.Ifthemasterserverfailsduringthisdelay,thencommittedandconfirmeddatacanbelost.

Nevertheless,wealsoofferautomaticfail-overwiththissetup.Contrarytothesynchronouscase,herethefail-overmanagementisdonefromoutsidetheArangoDBcluster.InafutureversionwemightmovethismanagementintothesupervisionprocessintheAgency,butasofnow,themanagementisdoneviatheMesosframeworkschedulerforArangoDB(seebelow).

ThegranularityofthereplicationisawholeArangoDBinstancewithalldatathatresidesonthatinstance,whichmeansthatyouneedtwiceasmanyinstancesaswithoutasynchronousreplication.Synchronousreplicationismoreflexibleinthatrespect,youcanhavesmallerandlargerinstances,andifonefails,thedatacanberebalancedacrosstheremainingones.

Microservicesandzeroadministation

ThedesignandcapabilitiesofArangoDBaregearedtowardsusageinmodernmicroservicearchitecturesofapplications.WiththeFoxxservicesitisveryeasytodeployadatacentricmicroservicewithinanArangoDBcluster.

Inaddition,onecandeploymultipleinstancesofArangoDBwithinthesameproject.Onepartoftheprojectmightneedascalabledocumentstore,anothermightneedagraphdatabase,andyetanothermightneedthefullpowerofamulti-modeldatabaseactuallymixingthevariousdatamodels.Thereareenormousefficiencybenefitstobereapedbybeingabletouseasingletechnologyforvariousrolesinaproject.

TosimplifylifeofthedevopsinsuchascenariowetryasmuchaspossibletouseazeroadministrationapproachforArangoDB.ArunningArangoDBclusterisresilientagainstfailuresandessentiallyrepairsitselfincaseoftemporaryfailures.Seethenextsectionforfurthercapabilitiesinthisdirection.

ApacheMesosintegration

Forthedistributedsetup,weusetheApacheMesosinfrastructurebydefault.ArangoDBisafullycertifiedpackageforDC/OSandcanthusbedeployedessentiallywithafewmouseclicksorasinglecommand,onceyouhaveanexistingDC/OScluster.ButevenonaplainApacheMesosclusteronecandeployArangoDBviaMarathonwithasingleAPIcallandsomeJSONconfiguration.

Theadvantageofthisapproachisthatwecannotonlyimplementtheinitialdeployment,butalsothelatermanagementofautomaticreplacementoffailedinstancesandthescalingoftheArangoDBcluster(triggeredmanuallyorevenautomatically).SinceallmanipulationsareeitherviathegraphicalwebUIorviaJSON/RESTcalls,onecanevenimplementauto-scalingveryeasily.

ADC/OSclusterisaverynaturalenvironmenttodeploymicroservicearchitectures,sinceitissoconvenienttodeployvariousservices,includingpotentiallymultipleArangoDBclusterinstanceswithinthesameDC/OScluster.Thebuilt-inservicediscoverymakesitextremelysimpletoconnectthevariousmicroservicesandMesosautomaticallytakescareofthedistributionanddeploymentofthevarioustasks.

SeetheDeploymentchapteranditssubsectionsforinstructions.

ItispossibletodeployanArangoDBclusterbysimplylaunchingabunchofDockercontainerswiththerightcommandlineoptionstolinkthemup,orevenonasinglemachinestartingmultipleArangoDBprocesses.Inthatcase,synchronousreplicationwillworkwithinthedeployedArangoDBcluster,andautomaticfail-overinthesensethatthedutiesofafailedserverwillautomaticallybeassignedtoanother,survivingone.However,sincetheArangoDBclustercannotwithinitselflaunchadditionalinstances,replacementoffailednodesisnotautomaticandscalingupanddownhastobemanagedmanually.Thisiswhywedonotrecommendthissetupforproductiondeployment.

Architecture

43

Differentdatamodelsandscalability

InthissectionwediscussscalabilityinthecontextofthedifferentdatamodelssupportedbyArangoDB.

Key/valuepairs

Thekey/valuestoredatamodelistheeasiesttoscale.InArangoDB,thisisimplementedinthesensethatadocumentcollectionalwayshasaprimarykey_keyattributeandintheabsenceoffurthersecondaryindexesthedocumentcollectionbehaveslikeasimplekey/valuestore.

Theonlyoperationsthatarepossibleinthiscontextaresinglekeylookupsandkey/valuepairinsertionsandupdates.If_keyistheonlyshardingattributethentheshardingisdonewithrespecttotheprimarykeyandalltheseoperationsscalelinearly.Iftheshardingisdoneusingdifferentshardkeys,thenalookupofasinglekeyinvolvesaskingallshardsandthusdoesnotscalelinearly.

Documentstore

Forthedocumentstorecaseeveninthepresenceofsecondaryindexesessentiallythesameargumentsapply,sinceanindexforashardedcollectionissimplythesameasalocalindexforeachshard.Therefore,singledocumentoperationsstillscalelinearlywiththesizeofthecluster,unlessaspecialshardingconfigurationmakeslookupsorwriteoperationsmoreexpensive.

ForadeeperanalysisofthistopicseethisblogpostinwhichgoodlinearscalabilityofArangoDBforsingledocumentoperationsisdemonstrated.

Complexqueriesandjoins

TheAQLquerylanguageallowscomplexqueries,usingmultiplecollections,secondaryindexesaswellasjoins.Inparticularwiththelatter,scalingcanbeachallenge,sinceifthedatatobejoinedresidesondifferentmachines,alotofcommunicationhastohappen.TheAQLqueryexecutionengineorganizesadatapipelineacrosstheclustertoputtogethertheresultsinthemostefficientway.Thequeryoptimizerisawareoftheclusterstructureandknowswhatdataiswhereandhowitisindexed.Therefore,itcanarriveataninformeddecisionaboutwhatpartsofthequeryoughttorunwhereinthecluster.

Nevertheless,forcertaincomplicatedjoins,therearelimitsastowhatcanbeachieved.

Graphdatabase

Graphdatabasesareparticularlygoodatqueriesongraphsthatinvolvepathsinthegraphofanaprioriunknownlength.Forexample,findingtheshortestpathbetweentwoverticesinagraph,orfindingallpathsthatmatchacertainpatternstartingatagivenvertexaresuchexamples.

However,iftheverticesandedgesalongtheoccurringpathsaredistributedacrossthecluster,thenalotofcommunicationisnecessarybetweennodes,andperformancesuffers.Toachievegoodperformanceatscale,itisthereforenecessarytogetthedistributionofthegraphdataacrosstheshardsintheclusterright.Mostofthetime,theapplicationdevelopersandusersofArangoDBknowbest,howtheirgraphsarastructured.Therefore,ArangoDBallowsuserstospecify,accordingtowhichattributesthegraphdataissharded.Ausefulfirststepisusuallytomakesurethattheedgesoriginatingatavertexresideonthesameclusternodeasthevertex.

Datamodels

44

https://mesosphere.com/blog/2015/11/30/arangodb-benchmark-dcos/

Limitations

ArangoDBhasnobuilt-inlimitationstohorizontalscalability.ThecentralresilientAgencywilleasilysustainhundredsofDBserversandcoordinators,andtheusualdatabaseoperationsworkcompletelydecentrallyanddonotrequireassistanceoftheAgency.

Likewise,thesupervisionprocessintheAgencycaneasilydealwithlotsofservers,sinceallitsactivitiesarenotperformancecritical.

Obviously,anArangoDBclusterislimitedbytheavailableresourcesofCPU,memory,diskandnetworkbandwidthandlatency.

Limitations

45

Datamodels&modelingThischapterintroducesArangoDB'scoreconceptsandcovers

itsdatamodel(ordatamodelsrespectively),theterminologyusedthroughoutthedatabasesystemandinthisdocumentation,aswellasaspectstoconsiderwhenmodelingyourdatatostrikeabalancebetweennaturaldatastructuresandgreatperformance

Youwillalsofindusageexamplesonhowtointeractwiththedatabasesystemusingarangosh,e.g.howtocreateanddropdatabases/collections,orhowtosave,update,replaceandremovedocuments.Youcandoallthisusingthewebinterfaceaswellandmaythereforeskipthesesectionsasbeginner.

Datamodels&modeling

46

Concepts

DatabaseInteraction

ArangoDBisadatabasethatservesdocumentstoclients.ThesedocumentsaretransportedusingJSONviaaTCPconnection,usingtheHTTPprotocol.ARESTAPIisprovidedtointeractwiththedatabasesystem.

ThewebinterfacethatcomeswithArangoDB,calledAardvark,providesgraphicaluserinterfacethatiseasytouse.Aninteractiveshell,calledArangosh,isalsoshipped.Inaddition,therearesocalleddriversthatmakeiteasytousethedatabasesysteminvariousenvironmentsandprogramminglanguages.AllthesetoolsusetheHTTPinterfaceoftheserverandremovethenecessitytorollownlow-levelcodeforbasiccommunicationinmostcases.

DatamodelThedocumentsyoucanstoreinArangoDBcloselyfollowtheJSONformat,althoughtheyarestoredinabinaryformatcalledVelocyPack.Adocumentcontainszeroormoreattributes,eachoftheseattributeshavingavalue.Avaluecaneitherbeanatomictype,i.e.number,string,booleanornull,oracompoundtype,i.e.anarrayorembeddeddocument/object.Arraysandsub-objectscancontainallofthesetypes,whichmeansthatarbitrarilynesteddatastructurescanberepresentedinasingledocument.

Documentsaregroupedintocollections.Acollectioncontainszeroormoredocuments.Ifyouarefamiliarwithrelationaldatabasemanagementsystems(RDBMS)thenitissafetocomparecollectionstotablesanddocumentstorows.ThedifferenceisthatinatraditionalRDBMS,youhavetodefinecolumnsbeforeyoucanstorerecordsinatable.Suchdefinitionsarealsoknownasschemas.ArangoDBisschema-less,whichmeansthatthereisnoneedtodefinewhatattributesadocumentcanhave.Everysingledocumentcanhaveacompletelydifferentstructureandstillbestoredtogetherwithotherdocumentsinasinglecollection.Inpractice,therewillbecommondenominatorsamongthedocumentsinacollection,butthedatabasesystemitselfdoesn'tforceyoutolimityourselftoacertaindatastructure.

Therearetwotypesofcollections:documentcollection(alsoreferedtoasvertexcollectionsinthecontextofgraphs)aswellasedgecollections.Edgecollectionsstoredocumentsaswell,buttheyincludetwospecialattributes,_fromand_to,whichareusedtocreaterelationsbetweendocuments.Usually,twodocuments(vertices)storedindocumentcollectionsarelinkedbyadocument(edge)storedinanedgecollection.ThisisArangoDB'sgraphdatamodel.Itfollowsthemathematicalconceptofadirected,labeledgraph,exceptthatedgesdon'tjusthavelabels,butarefull-blowndocuments.

Collectionsexistinsideofdatabases.Therecanbeoneormanydatabases.Differentdatabasesareusuallyusedformultitenantsetups,asthedatainsidethem(collections,documentsetc.)isisolatedfromoneanother.Thedefaultdatabase_systemisspecial,becauseitcannotberemoved.Databaseusersaremanagedinthisdatabase,andtheircredentialsarevalidforalldatabasesofaserverinstance.

DataRetrievalQueriesareusedtofilterdocumentsbasedoncertaincriteria,tocomputenewdata,aswellastomanipulateordeleteexistingdocuments.Queriescanbeassimpleasa"querybyexample"orascomplexas"joins"usingmanycollectionsortraversinggraphstructures.TheyarewrittenintheArangoDBQueryLanguage(AQL).

Cursorsareusedtoiterateovertheresultofqueries,sothatyougeteasilyprocessablebatchesinsteadofonebighunk.

Indexesareusedtospeedupsearches.Therearevarioustypesofindexes,suchashashindexesandgeoindexes.

Concepts

47

https://en.wikipedia.org/wiki/JSON

https://en.wikipedia.org/wiki/Representational_state_transfer

https://arangodb.com/downloads/arangodb-drivers/

https://github.com/arangodb/velocypack#readme

HandlingDatabasesThisisanintroductiontomanagingdatabasesinArangoDBfromwithinJavaScript.

WhenyouhaveanestablishedconnectiontoArangoDB,thecurrentdatabasecanbechangedexplicitlyusingthedb._useDatabase()method.Thiswillswitchtothespecifieddatabase(provideditexistsandtheusercanconnecttoit).Fromthispointon,anyfollowingactioninthesameshellorconnectionwillusethespecifieddatabase,unlessotherwisespecified.

Note:Ifthedatabaseischanged,clientdriversneedtostorethecurrentdatabasenameontheirside,too.ThisisbecauseconnectionsinArangoDBdonotcontainanystateinformation.AllstateinformationiscontainedintheHTTPrequest/responsedata.

Toconnecttoaspecificdatabaseafterarangoshhasstartedusethecommanddescribedabove.Itisalsopossibletospecifyadatabasenamewheninvokingarangosh.Forthispurpose,usethecommand-lineparameter--server.database,e.g.

>arangosh--server.databasetest

Pleasenotethatcommands,actions,scriptsorAQLqueriesshouldneveraccessmultipledatabases,eveniftheyexist.TheonlyintendedandsupportedwayinArangoDBistouseonedatabaseatatimeforacommand,anaction,ascriptoraquery.Operationsstartedinonedatabasemustnotswitchthedatabaselaterandcontinueoperatinginanother.

Databases

48


DatabaseMethods

ThefollowingmethodsareavailabletomanagedatabasesviaJavaScript.Pleasenotethatseveralofthesemethodscanbeusedfromthe_systemdatabaseonly.

Name

returnthedatabasenamedb._name()

Returnsthenameofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._name();

_system

ID

returnthedatabaseiddb._id()

Returnstheidofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._id();

1

Path

returnthepathtodatabasefilesdb._path()

Returnsthefilesystempathofthecurrentdatabaseasastring.

Examples

arangosh>require("@arangodb").db._path();

/tmp/arangosh_aFcSh5/tmp-9239-3002331047/data/databases/database-1

isSystem

returnthedatabasetypedb._isSystem()

Returnswhetherthecurrentlyuseddatabaseisthe_systemdatabase.Thesystemdatabasehassomespecialprivilegesandproperties,forexample,databasemanagementoperationssuchascreateordropcanonlybeexecutedfromwithinthisdatabase.Additionally,the_systemdatabaseitselfcannotbedropped.

UseDatabase

changethecurrentdatabasedb._useDatabase(name)

Changesthecurrentdatabasetothedatabasespecifiedbyname.Notethatthedatabasespecifiedbynamemustalreadyexist.

Changingthedatabasemightbedisallowedinsomecontexts,forexampleserver-sideactions(includingFoxx).


49

Whenperformingthiscommandfromarangosh,thecurrentcredentials(usernameandpassword)willbere-used.Thesecredentialsmightnotbevalidtoconnecttothedatabasespecifiedbyname.Additionally,thedatabaseonlybeaccessedfromcertainendpointsonly.Inthiscase,switchingthedatabasemightnotwork,andtheconnection/sessionshouldbeclosedandrestartedwithdifferentusernameandpasswordcredentialsand/orendpointdata.

ListDatabases

returnthelistofallexistingdatabasesdb._databases()

Returnsthelistofalldatabases.Thismethodcanonlybeusedfromwithinthe_systemdatabase.

CreateDatabase

createanewdatabasedb._createDatabase(name,options,users)

Createsanewdatabasewiththenamespecifiedbyname.Therearerestrictionsfordatabasenames(seeDatabaseNames).

Notethatevenifthedatabaseiscreatedsuccessfully,therewillbenochangeintothecurrentdatabasetothenewdatabase.Changingthecurrentdatabasemustexplicitlyberequestedbyusingthedb._useDatabasemethod.

Theoptionsattributecurrentlyhasnomeaningandisreservedforfutureuse.

Theoptionalusersattributecanbeusedtocreateinitialusersforthenewdatabase.Ifspecified,itmustbealistofuserobjects.Eachuserobjectcancontainthefollowingattributes:

username:theusernameasastring.Thisattributeismandatory.passwd:theuserpasswordasastring.Ifnotspecified,thenitdefaultstoanemptystring.active:abooleanflagindicatingwhethertheuseraccountshouldbeactiveornot.Thedefaultvalueistrue.extra:anoptionalJSONobjectwithextrauserinformation.ThedatacontainedinextrawillbestoredfortheuserbutnotbeinterpretedfurtherbyArangoDB.

Ifnoinitialusersarespecified,adefaultuserrootwillbecreatedwithanemptystringpassword.ThisensuresthatthenewdatabasewillbeaccessibleviaHTTPafteritiscreated.

Youcancreateusersinadatabaseifnoinitialuserisspecified.Switchintothenewdatabase(usernameandpasswordmustbeidenticaltothecurrentsession)andaddormodifyuserswiththefollowingcommands.

require("@arangodb/users").save(username,password,true);

require("@arangodb/users").update(username,password,true);

require("@arangodb/users").remove(username);

Alternatively,youcanspecifyuserdatadirectly.Forexample:

db._createDatabase("newDB",{},[{username:"newUser",passwd:"123456",active:true}])

Thosemethodscanonlybeusedfromwithinthe_systemdatabase.

DropDatabase

dropanexistingdatabasedb._dropDatabase(name)

Dropsthedatabasespecifiedbyname.Thedatabasespecifiedbynamemustexist.

Note:Droppingdatabasesisonlypossiblefromwithinthe_systemdatabase.The_systemdatabaseitselfcannotbedropped.

Databasesaredroppedasynchronously,andwillbephysicallyremovedifallclientshavedisconnectedandreferenceshavebeengarbage-collected.

Enginestatistics

retrievestatisticsrelatedtothestorageengine-rocksdbdb._engineStats()

Returnssomestatisticsrelatedtostorageengineactivity,includingfiguresaboutdatasize,cacheusage,etc.


50

Note:CurrentlythisonlyproducesusefuloutputfortheRocksDBengine.


51

NotesaboutDatabasesPleasekeepinmindthateachdatabasecontainsitsownsystemcollections,whichneedtobesetupwhenadatabaseiscreated.Thiswillmakethecreationofadatabasetakeawhile.

Replicationisconfiguredonaper-databaselevel,meaningthatanyreplicationloggingorapplyingforanewdatabasemustbeconfiguredexplicitlyafteranewdatabasehasbeencreated.

Foxxapplicationsarealsoavailableonlyinthecontextofthedatabasetheyhavebeeninstalledin.AnewdatabasewillonlyprovideaccesstothesystemapplicationsshippedwithArangoDB(thatisthewebinterfaceatthemoment)andnootherFoxxapplicationsuntiltheyareexplicitlyinstalledfortheparticulardatabase.

NotesaboutDatabases

52

JavaScriptInterfacetoCollectionsThisisanintroductiontoArangoDB'sinterfaceforcollectionsandhowtohandlecollectionsfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.

Themostimportantcallisthecalltocreateanewcollection.

AddressofaCollection

AllcollectionsinArangoDBhaveauniqueidentifierandauniquename.ArangoDBinternallyusesthecollection'suniqueidentifiertolookupcollections.Thisidentifier,however,ismanagedbyArangoDBandtheuserhasnocontroloverit.Inordertoallowuserstousetheirownnames,eachcollectionalsohasauniquenamewhichisspecifiedbytheuser.Toaccessacollectionfromtheuserperspective,thecollectionnameshouldbeused,i.e.:

Collection

db._collection(collection-name)

Acollectioniscreatedbya"db._create"call.

Forexample:Assumethatthecollectionidentifieris7254820andthenameisdemo,thenthecollectioncanbeaccessedas:

db._collection("demo")

Ifnocollectionwithsuchanameexists,thennullisreturned.

Thereisashort-cutthatcanbeusedfornon-systemcollections:

Collectionname

db.collection-name

Thiscallwilleitherreturnthecollectionnameddb.collection-nameorcreateanewonewiththatnameandasetofdefaultproperties.

Note:Creatingacollectionontheflyusingdb.collection-nameisnotrecommendanddoesnotworkinarangosh.Tocreateanewcollection,pleaseuse

Create

db._create(collection-name)

Thiscallwillcreateanewcollectioncalledcollection-name.ThismethodisadatabasemethodandisdocumentedindetailatDatabaseMethods

Synchronousreplication

StartinginArangoDB3.0,thedistributedversionofferssynchronousreplication,whichmeansthatthereistheoptiontoreplicatealldataautomaticallywithintheArangoDBcluster.Thisisconfiguredforshardedcollectionsonapercollectionbasisbyspecifyinga"replicationfactor"whenthecollectioniscreated.Areplicationfactorofkmeansthataltogetherkcopiesofeachshardarekeptintheclusteronkdifferentservers,andarekeptinsync.Thatis,everywriteoperationisautomaticallyreplicatedonallcopies.

Thisisorganisedusingaleader/followermodel.Atalltimes,oneoftheserversholdingreplicasforashardis"theleader"andallothersare"followers",thisconfigurationisheldintheAgency(seeScalabilityfordetailsoftheArangoDBclusterarchitecture).Everywriteoperationissenttotheleaderbyoneofthecoordinators,andthenreplicatedtoallfollowersbeforetheoperationisreportedtohavesucceeded.Theleaderkeepsarecordofwhichfollowersarecurrentlyinsync.Incaseofnetworkproblemsorafailureofafollower,aleadercanandwilldropafollowertemporarilyafter3seconds,suchthatservicecanresume.Induecourse,thefollowerwillautomaticallyresynchronizewiththeleadertorestoreresilience.

Collections

53

Ifaleaderfails,theclusterAgencyautomaticallyinitiatesafailoverroutineafteraround15seconds,promotingoneofthefollowerstoleader.Theotherfollowers(andtheformerleader,whenitcomesback),automaticallyresynchronizewiththenewleadertorestoreresilience.Usually,thiswholefailoverprocedurecanbehandledtransparentlyforthecoordinator,suchthattheusercodedoesnotevenseeanerrormessage.

Obviously,thisfaulttolerancecomesatacostofincreasedlatency.Eachwriteoperationneedsanadditionalnetworkroundtripforthesynchronousreplicationofthefollowers,butallreplicationoperationstoallfollowershappenconcurrently.Thisis,whythedefaultreplicationfactoris1,whichmeansnoreplication.

Fordetailsonhowtoswitchonsynchronousreplicationforacollection,seethedatabasemethoddb._create(collection-name)inthesectionaboutDatabaseMethods.

Collections

54

CollectionMethods

Drop

dropsacollectioncollection.drop(options)

Dropsacollectionandallitsindexesanddata.Inordertodropasystemcollection,anoptionsobjectwithattributeisSystemsettotruemustbespecified.

Note:droppingacollectioninacluster,whichisprototypeforsharinginothercollectionsisprohibited.Inordertobeabletodropsuchacollection,alldependentcollectionsmustbedroppedfirst.

Examples

arangosh>col=db.example;

[ArangoCollection15512,"example"(typedocument,statusloaded)]

arangosh>col.drop();

arangosh>col;

[ArangoCollection15512,"example"(typedocument,statusdeleted)]

arangosh>col=db._example;

[ArangoCollection15516,"_example"(typedocument,statusloaded)]

arangosh>col.drop({isSystem:true});

arangosh>col;

[ArangoCollection15516,"_example"(typedocument,statusdeleted)]

Truncate

truncatesacollectioncollection.truncate()

Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.

Examples

Truncatesacollection:


arangosh>col.save({"Hello":"World"});

arangosh>col.count();

arangosh>col.truncate();


showexecutionresults

Properties

getsorsetsthepropertiesofacollectioncollection.properties()

Returnsanobjectcontainingallcollectionproperties.

waitForSync:Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.

journalSize:Thesizeofthejournalinbytes.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

isVolatile:IftruethenthecollectiondatawillbekeptinmemoryonlyandArangoDBwillnotwriteorsyncthedatatodisk.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

CollectionMethods

55

keyOptions(optional)additionaloptionsforkeygeneration.ThisisaJSONarraycontainingthefollowingattributes(note:someoftheattributesareoptional):

type:thetypeofthekeygeneratorusedforthecollection.allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.

indexBuckets:numberofbucketsintowhichindexesusingahashtablearesplit.Thedefaultis16andthisnumberhastobeapowerof2andlessthanorequalto1024.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

Forverylargecollectionsoneshouldincreasethistoavoidlongpauseswhenthehashtablehastobeinitiallybuiltorresized,sincebucketsareresizedindividuallyandcanbeinitiallybuiltinparallel.Forexample,64mightbeasensiblevalueforacollectionwith100000000documents.Currently,onlytheedgeindexrespectsthisvalue,butotherindextypesmightfollowinfutureArangoDBversions.Changes(seebelow)areappliedwhenthecollectionisloadedthenexttime.

Inaclustersetup,theresultwillalsocontainthefollowingattributes:

numberOfShards:thenumberofshardsofthecollection.

shardKeys:containsthenamesofdocumentattributesthatareusedtodeterminethetargetshardfordocuments.

replicationFactor:determineshowmanycopiesofeachshardarekeptondifferentDBServers.

collection.properties(properties)

Changesthecollectionproperties.propertiesmustbeanobjectwithoneormoreofthefollowingattribute(s):



indexBuckets:Seeabove,changesareonlyappliedwhenthecollectionisloadedthenexttime.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

replicationFactor:ChangethenumberofshardcopieskeptondifferentDBServers,validvaluesareintegernumbersintherangeof1-10(Clusteronly)

Note:itisnotpossibletochangethejournalsizeafterthejournalordatafilehasbeencreated.Changingthisparameterwillonlyeffectnewlycreatedjournals.Alsonotethatyoucannotlowerthejournalsizetolessthensizeofthelargestdocumentalreadystoredinthecollection.

Note:someothercollectionproperties,suchastype,isVolatile,orkeyOptionscannotbechangedoncethecollectioniscreated.

Examples

Readallproperties

arangosh>db.example.properties();

showexecutionresultsChangeaproperty

arangosh>db.example.properties({waitForSync:true});


Figures

returnsthefiguresofacollectioncollection.figures()

Returnsanobjectcontainingstatisticsaboutthecollection.Note:Retrievingthefigureswillalwaysloadthecollectionintomemory.

alive.count:Thenumberofcurrentlyactivedocumentsinalldatafilesandjournalsofthecollection.Documentsthatarecontainedin

CollectionMethods

56

thewrite-aheadlogonlyarenotreportedinthisfigure.

alive.size:Thetotalsizeinbytesusedbyallactivedocumentsofthecollection.Documentsthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.dead.count:Thenumberofdeaddocuments.Thisincludesdocumentversionsthathavebeendeletedorreplacedbyanewerversion.Documentsdeletedorreplacedthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.dead.size:Thetotalsizeinbytesusedbyalldeaddocuments.dead.deletion:Thetotalnumberofdeletionmarkers.Deletionmarkersonlycontainedinthewrite-aheadlogarenotreportinginthisfigure.datafiles.count:Thenumberofdatafiles.datafiles.fileSize:Thetotalfilesizeofdatafiles(inbytes).journals.count:Thenumberofjournalfiles.journals.fileSize:Thetotalfilesizeofthejournalfiles(inbytes).compactors.count:Thenumberofcompactorfiles.compactors.fileSize:Thetotalfilesizeofthecompactorfiles(inbytes).shapefiles.count:Thenumberofshapefiles.Thisvalueisdeprecatedandkeptforcompatibilityreasonsonly.Thevaluewillalwaysbe0sinceArangoDB2.0andhigher.shapefiles.fileSize:Thetotalfilesizeoftheshapefiles.Thisvalueisdeprecatedandkeptforcompatibilityreasonsonly.Thevaluewillalwaysbe0inArangoDB2.0andhigher.shapes.count:Thetotalnumberofshapesusedinthecollection.Thisincludesshapesthatarenotinuseanymore.Shapesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.shapes.size:Thetotalsizeofallshapes(inbytes).Thisincludesshapesthatarenotinuseanymore.Shapesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.attributes.count:Thetotalnumberofattributesusedinthecollection.Note:thevalueincludesdataofattributesthatarenotinuseanymore.Attributesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.attributes.size:Thetotalsizeoftheattributedata(inbytes).Note:thevalueincludesdataofattributesthatarenotinuseanymore.Attributesthatarecontainedinthewrite-aheadlogonlyarenotreportedinthisfigure.indexes.count:Thetotalnumberofindexesdefinedforthecollection,includingthepre-definedindexes(e.g.primaryindex).indexes.size:Thetotalmemoryallocatedforindexesinbytes.lastTick:Thetickofthelastmarkerthatwasstoredinajournalofthecollection.Thismightbe0ifthecollectiondoesnotyethaveajournal.uncollectedLogfileEntries:Thenumberofmarkersinthewrite-aheadlogforthiscollectionthathavenotbeentransferredtojournalsordatafiles.documentReferences:ThenumberofreferencestodocumentsindatafilesthatJavaScriptcodecurrentlyholds.Thisinformationcanbeusedfordebuggingcompactionandunloadissues.waitingFor:Anoptionalstringvaluethatcontainsinformationaboutwhichobjecttypeisattheheadofthecollection'scleanupqueue.Thisinformationcanbeusedfordebuggingcompactionandunloadissues.compactionStatus.time:Thepointintimethecompactionforthecollectionwaslastexecuted.Thisinformationcanbeusedfordebuggingcompactionissues.compactionStatus.message:Theactionthatwasperformedwhenthecompactionwaslastrunforthecollection.Thisinformationcanbeusedfordebuggingcompactionissues.

Note:collectiondatathatarestoredinthewrite-aheadlogonlyarenotreportedintheresults.Whenthewrite-aheadlogiscollected,documentsmightbeaddedtojournalsanddatafilesofthecollection,whichmaymodifythefiguresofthecollection.AlsonotethatwaitingForandcompactionStatusmaybeemptywhencalledonacoordinatorinacluster.

Additionally,thefilesizesofcollectionandindexparameterJSONfilesarenotreported.Thesefilesshouldnormallyhaveasizeofafewbyteseach.PleasealsonotethatthefileSizevaluesarereportedinbytesandreflectthelogicalfilesizes.Somefilesystemsmayuseoptimisations(e.g.sparsefiles)sothattheactualphysicalfilesizeissomewhatdifferent.Directoriesandsub-directoriesmayalsorequirespaceinthefilesystem,butthisspaceisnotreportedinthefileSizeresults.

Thatmeansthatthefiguresreporteddonotreflecttheactualdiskusageofthecollectionwith100%accuracy.TheactualdiskusageofacollectionisnormallyslightlyhigherthanthesumofthereportedfileSizevalues.StillthesumofthefileSizevaluescanstillbeusedasalowerboundapproximationofthediskusage.

Examples

arangosh>db.demo.figures()

CollectionMethods

57


Load

loadsacollectioncollection.load()

Loadsacollectionintomemory.

Note:clustercollectionsareloadedatalltimes.

Examples



arangosh>col.load();

arangosh>col;


Revision

returnstherevisionidofacollectioncollection.revision()

Returnstherevisionidofthecollection

Therevisionidisupdatedwhenthedocumentdataismodified,eitherbyinserting,deleting,updatingorreplacingdocumentsinit.

Therevisionidofacollectioncanbeusedbyclientstocheckwhetherdatainacollectionhaschangedorifitisstillunmodifiedsinceapreviousfetchoftherevisionid.

Therevisionidreturnedisastringvalue.Clientsshouldtreatthisvalueasanopaquestring,andonlyuseitforequality/non-equalitycomparisons.

Path

returnsthephysicalpathofthecollectioncollection.path()

Thepathoperationreturnsastringwiththephysicalstoragepathforthecollectiondata.

Note:thismethodwillreturnnothingmeaningfulinacluster.Inasingle-serverArangoDB,thismethodwillonlyreturnmeaningfuldatafortheMMFilesengine.

Checksum

calculatesachecksumforthedatainacollectioncollection.checksum(withRevisions,withData)

Thechecksumoperationcalculatesanaggregatehashvalueforalldocumentkeyscontainedincollectioncollection.

IftheoptionalargumentwithRevisionsissettotrue,thentherevisionidsofthedocumentsarealsoincludedinthehashcalculation.

IftheoptionalargumentwithDataissettotrue,thenalluser-defineddocumentattributesarealsochecksummed.Includingthedocumentdatainchecksummingwillmakethecalculationslower,butismoreaccurate.

ThechecksumcalculationalgorithmchangedinArangoDB3.0,sochecksumsfrom3.0andearlierversionsforthesamedatawilldiffer.

Note:thismethodisnotavailableinacluster.

Unload

unloadsacollectioncollection.unload()

Startsunloadingacollectionfrommemory.Notethatunloadingisdeferreduntilallqueryhavefinished.

Note:clustercollectionscannotbeunloaded.

Examples

CollectionMethods

58



arangosh>col.unload();

arangosh>col;

[ArangoCollection7427,"example"(typedocument,statusunloaded)]

Rename

renamesacollectioncollection.rename(new-name)

Renamesacollectionusingthenew-name.Thenew-namemustnotalreadybeusedforadifferentcollection.new-namemustalsobeavalidcollectionname.Formoreinformationonvalidcollectionnamespleaserefertothenamingconventions.

Ifrenamingfailsforanyreason,anerroristhrown.Ifrenamingthecollectionsucceeds,thenthecollectionisalsorenamedinallgraphdefinitionsinsidethe_graphscollectioninthecurrentdatabase.

Note:thismethodisnotavailableinacluster.

Examples

arangosh>c=db.example;


arangosh>c.rename("better-example");

arangosh>c;

[ArangoCollection15669,"better-example"(typedocument,statusloaded)]

Rotate

rotatesthecurrentjournalofacollectioncollection.rotate()

Rotatesthecurrentjournalofacollection.Thisoperationmakesthecurrentjournalofthecollectionaread-onlydatafilesoitmaybecomeacandidateforgarbagecollection.Ifthereiscurrentlynojournalavailableforthecollection,theoperationwillfailwithanerror.

Note:thismethodisspecificfortheMMFilesstorageengine,andthereitisnotavailableinacluster.

Note:pleasenotethatyouneedappropriateuserpermissionstoexecutethis.

TodotherenamecollectionsinfirstplaceyouneedtohaveadministrativerightsonthedatabaseTohaveaccesstotheresultingrenamedcollectionyoueitherneedtohaveaccesstoallcollectionsofthatdatabase(*)oramainsystemadministratorhastogiveyouaccesstothenewlynamedone.

CollectionMethods

59

DatabaseMethods

Collection

returnsasinglecollectionornulldb._collection(collection-name)

Returnsthecollectionwiththegivennameornullifnosuchcollectionexists.

db._collection(collection-identifier)

Returnsthecollectionwiththegivenidentifierornullifnosuchcollectionexists.Accessingcollectionsbyidentifierisdiscouragedforendusers.Endusersshouldaccesscollectionsusingthecollectionname.

Examples

Getacollectionbyname:

arangosh>db._collection("demo");

[ArangoCollection92,"demo"(typedocument,statusloaded)]

Getacollectionbyid:

arangosh>db._collection(123456);

[ArangoCollection123456,"demo"(typedocument,statusloaded)]

Unknowncollection:

arangosh>db._collection("unknown");

null

Create

createsanewdocumentoredgecollectiondb._create(collection-name)

Createsanewdocumentcollectionnamedcollection-name.Ifthecollectionnamealreadyexistsorifthenameformatisinvalid,anerroristhrown.Formoreinformationonvalidcollectionnamespleaserefertothenamingconventions.

db._create(collection-name,properties)

propertiesmustbeanobjectwiththefollowingattributes:

waitForSync(optional,defaultfalse):Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.

journalSize(optional,defaultisaconfigurationparameter:Themaximalsizeofajournalordatafile.Notethatthisalsolimitsthemaximalsizeofasingleobject.Mustbeatleast1MB.

isSystem(optional,defaultisfalse):Iftrue,createasystemcollection.Inthiscasecollection-nameshouldstartwithanunderscore.Endusersshouldnormallycreatenon-systemcollectionsonly.APIimplementorsmayberequiredtocreatesystemcollectionsinveryspecialoccasions,butnormallyaregularcollectionwilldo.

isVolatile(optional,defaultisfalse):Iftruethenthecollectiondataiskeptin-memoryonlyandnotmadepersistent.Unloadingthecollectionwillcausethecollectiondatatobediscarded.Stoppingorre-startingtheserverwillalsocausefulllossofdatainthecollection.Thecollectionitselfwillremainhowever(onlythedataisvolatile).SettingthisoptionwillmaketheresultingcollectionbeslightlyfasterthanregularcollectionsbecauseArangoDBdoesnotenforceanysynchronizationtodiskanddoesnotcalculateanyCRCchecksumsfordatafiles(astherearenodatafiles).ThisoptionismeaningfulfortheMMFilesstorageengineonly.

keyOptions(optional):additionaloptionsforkeygeneration.Ifspecified,thenkeyOptionsshouldbeaJSONobjectcontainingthefollowingattributes(note:someofthemareoptional):

type:specifiesthetypeofthekeygenerator.Thecurrentlyavailablegeneratorsaretraditionalandautoincrement.(note:autoincrementiscurrentlyonlysupportedfornon-shardedcollections)

DatabaseMethods

60

allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.

numberOfShards(optional,defaultis1):inacluster,thisvaluedeterminesthenumberofshardstocreateforthecollection.Inasingleserversetup,thisoptionismeaningless.

shardKeys(optional,defaultis["_key"]):inacluster,thisattributedetermineswhichdocumentattributesareusedtodeterminethetargetshardfordocuments.Documentsaresenttoshardsbasedonthevaluestheyhaveintheirshardkeyattributes.Thevaluesofallshardkeyattributesinadocumentarehashed,andthehashvalueisusedtodeterminethetargetshard.Notethatvaluesofshardkeyattributescannotbechangedonceset.Thisoptionismeaninglessinasingleserversetup.

Whenchoosingtheshardkeys,onemustbeawareofthefollowingrulesandlimitations:InashardedcollectionwithmorethanonesharditisnotpossibletosetupauniqueconstraintonanattributethatisnottheoneandonlyshardkeygiveninshardKeys.Thisisbecauseenforcingauniqueconstraintwouldotherwisemakeaglobalindexnecessaryorneedextensivecommunicationforeverysinglewriteoperation.Furthermore,if_keyisnottheoneandonlyshardkey,thenitisnotpossibletosetthe_keyattributewheninsertingadocument,providedthecollectionhasmorethanoneshard.Again,thisisbecausethedatabasehastoenforcetheuniqueconstraintonthe_keyattributeandthiscanonlybedoneefficientlyifthisistheonlyshardkeybydelegatingtotheindividualshards.

replicationFactor(optional,defaultis1):inacluster,thisattributedetermineshowmanycopiesofeachshardarekeptondifferentDBServers.Thevalue1meansthatonlyonecopy(nosynchronousreplication)iskept.Avalueofkmeansthatk-1replicasarekept.AnytwocopiesresideondifferentDBServers.Replicationbetweenthemissynchronous,thatis,everywriteoperationtothe"leader"copywillbereplicatedtoall"follower"replicas,beforethewriteoperationisreportedsuccessful.

Ifaserverfails,thisisdetectedautomaticallyandoneoftheserversholdingcopiestakeover,usuallywithoutanerrorbeingreported.

WhenusingtheEnterpriseversionofArangoDBthereplicationFactormaybesetto"satellite"makingthecollectionlocallyjoinableoneverydatabaseserver.ThisreducesthenumberofnetworkhopsdramaticallywhenusingjoinsinAQLatthecostsofreducedwriteperformanceonthesecollections.

distributeShardsLikedistributetheshardsofthiscollectioncloningthesharddistributionofanother.IfthisvalueissetitwillcopyreplicationFactorandnumberOfShardsfromtheothercollection,theattributesinthiscollectionwillbeignoredandcanbeommited.

db._create(collection-name,properties,type)

Specifiestheoptionaltypeofthecollection,itcaneitherbedocumentoredge.Ondefaultitisdocument.Insteadofgivingatypeyoucanalsousedb._createEdgeCollectionordb._createDocumentCollection.

db._create(collection-name,properties[,type],options)

Asanoptionalthird(ifthetypestringisbeingomitted)orfourthparameteryoucanspecifyanoptionaloptionsmapthatcontrolshowtheclusterwillcreatethecollection.Theseoptionsareonlyrelevantatcreationtimeandwillnotbepersisted:

waitForSyncReplication(default:true)Whenenabledtheserverwillonlyreportsuccessbacktotheclientifallreplicashavecreatedthecollection.Settofalseifyouwantfasterserverresponsesanddon'tcareaboutfullreplication.

enforceReplicationFactor(default:true)Whenenabledwhichmeanstheserverwillcheckifthereareenoughreplicasavailableatcreationtimeandbailoutotherwise.Settofalsetodisablethisextracheck.

Examples

Withdefaults:

arangosh>c=db._create("users");

arangosh>c.properties();

showexecutionresultsWithproperties:

arangosh>c=db._create("users",{waitForSync:true,

DatabaseMethods

61

........>journalSize:1024*1204});

arangosh>c.properties();

showexecutionresultsWithakeygenerator:

arangosh>db._create("users",

........>{keyOptions:{type:"autoincrement",offset:10,increment:5}});

arangosh>db.users.save({name:"user1"});



showexecutionresultsWithaspecialkeyoption:

arangosh>db._create("users",{keyOptions:{allowUserKeys:false}});


arangosh>db.users.save({name:"user2",_key:"myuser"});


showexecutionresultscreatesanewedgecollectiondb._createEdgeCollection(collection-name)

Createsanewedgecollectionnamedcollection-name.Ifthecollectionnamealreadyexistsanerroristhrown.ThedefaultvalueforwaitForSyncisfalse.

db._createEdgeCollection(collection-name,properties)

propertiesmustbeanobjectwiththefollowingattributes:

waitForSync(optional,defaultfalse):Iftruecreatingadocumentwillonlyreturnafterthedatawassyncedtodisk.journalSize(optional,defaultis"configurationparameter"):Themaximalsizeofajournalordatafile.Notethatthisalsolimitsthemaximalsizeofasingleobjectandmustbeatleast1MB.

createsanewdocumentcollectiondb._createDocumentCollection(collection-name)

Createsanewdocumentcollectionnamedcollection-name.Ifthedocumentnamealreadyexistsanderroristhrown.

AllCollections

returnsallcollectionsdb._collections()

Returnsallcollectionsofthegivendatabase.

Examples

arangosh>db._collections();


CollectionName

selectsacollectionfromthevocbasedb.collection-name

Returnsthecollectionwiththegivencollection-name.Ifnosuchcollectionexists,createacollectionnamedcollection-namewiththedefaultproperties.

Examples

arangosh>db.example;

DatabaseMethods

62


Drop

dropsacollectiondb._drop(collection)

Dropsacollectionandallitsindexesanddata.

db._drop(collection-identifier)

Dropsacollectionidentifiedbycollection-identifierwithallitsindexesanddata.Noerroristhrownifthereisnosuchcollection.

db._drop(collection-name)

Dropsacollectionnamedcollection-nameandallitsindexes.Noerroristhrownifthereisnosuchcollection.

db._drop(collection-name,options)

Inordertodropasystemcollection,onemustspecifyanoptionsobjectwithattributeisSystemsettotrue.Otherwiseitisnotpossibletodropsystemcollections.

Note:clustercollection,whichareprototypesforcollectionswithdistributeShardsLikeparameter,cannotbedropped.

Examples

Dropsacollection:



arangosh>db._drop(col);

arangosh>col;


Dropsacollectionidentifiedbyname:



arangosh>db._drop("example");

arangosh>col;

[ArangoCollection15453,"example"(typedocument,statusdeleted)]

Dropsasystemcollection

arangosh>col=db._example;

[ArangoCollection15457,"_example"(typedocument,statusloaded)]

arangosh>db._drop("_example",{isSystem:true});

arangosh>col;

[ArangoCollection15457,"_example"(typedocument,statusdeleted)]

Truncate

truncatesacollectiondb._truncate(collection)

Truncatesacollection,removingalldocumentsbutkeepingallitsindexes.

db._truncate(collection-identifier)

Truncatesacollectionidentifiedbycollection-identified.Noerroristhrownifthereisnosuchcollection.

db._truncate(collection-name)

Truncatesacollectionnamedcollection-name.Noerroristhrownifthereisnosuchcollection.

Examples

DatabaseMethods

63

Truncatesacollection:




arangosh>db._truncate(col);


showexecutionresultsTruncatesacollectionidentifiedbyname:




arangosh>db._truncate("example");



DatabaseMethods

64

DocumentsThisisanintroductiontoArangoDB'sinterfaceforworkingwithdocumentsfromtheJavaScriptshellarangoshorinJavaScriptcodeintheserver.ForotherlanguagesseethecorrespondinglanguageAPI.

BasicsandTerminology:sectiononthebasicapproachCollectionMethods:detailedAPIdescriptionforcollectionobjectsDatabaseMethods:detailedAPIdescriptionfordatabaseobjects

Documents

65

BasicsandTerminologyDocumentsinArangoDBareJSONobjects.Theseobjectscanbenested(toanydepth)andmaycontainlists.Eachdocumenthasauniqueprimarykeywhichidentifiesitwithinitscollection.Furthermore,eachdocumentisuniquelyidentifiedbyitsdocumenthandleacrossallcollectionsinthesamedatabase.Differentrevisionsofthesamedocument(identifiedbyitshandle)canbedistinguishedbytheirdocumentrevision.Anytransactiononlyeverseesasinglerevisionofadocument.Forexample:

{

"_id":"myusers/3456789",

"_key":"3456789",

"_rev":"14253647",

"firstName":"John",

"lastName":"Doe",

"address":{

"street":"RoadToNowhere1",

"city":"Gotham"

},

"hobbies":[




]

}

Alldocumentscontainspecialattributes:thedocumenthandleisstoredasastringin_id,thedocument'sprimarykeyin_keyandthedocumentrevisionin_rev.Thevalueofthe_keyattributecanbespecifiedbytheuserwhencreatingadocument._idand_keyvaluesareimmutableoncethedocumenthasbeencreated.The_revvalueismaintainedbyArangoDBautomatically.

DocumentHandle

Adocumenthandleuniquelyidentifiesadocumentinthedatabase.Itisastringandconsistsofthecollection'snameandthedocumentkey(_keyattribute)separatedby/.

DocumentKey

Adocumentkeyuniquelyidentifiesadocumentinthecollectionitisstoredin.Itcanandshouldbeusedbyclientswhenspecificdocumentsarequeried.Thedocumentkeyisstoredinthe_keyattributeofeachdocument.ThekeyvaluesareautomaticallyindexedbyArangoDBinacollection'sprimaryindex.Thuslookingupadocumentbyitskeyisafastoperation.The_keyvalueofadocumentisimmutableoncethedocumenthasbeencreated.Bydefault,ArangoDBwillauto-generateadocumentkeyifno_keyattributeisspecified,andusetheuser-specified_keyotherwise.Thegenerated_keyisguaranteedtobeuniqueinthecollectionitwasgeneratedfor.Thisalsoappliestoshardedcollectionsinacluster.Itcan'tbeguaranteedthatthe_keyisuniquewithinadatabaseoracrossawholenodeorinstancehowever.

Thisbehaviorcanbechangedonaper-collectionlevelbycreatingcollectionswiththekeyOptionsattribute.

UsingkeyOptionsitispossibletodisallowuser-specifiedkeyscompletely,ortoforceaspecificregimeforauto-generatingthe_keyvalues.

DocumentRevision

AsArangoDBsupportsMVCC(MultipleVersionConcurrencyControl),documentscanexistinmorethanonerevision.ThedocumentrevisionistheMVCCtokenusedtospecifyaparticularrevisionofadocument(identifiedbyits_id).Itisastringvaluethatcontained(uptoArangoDB3.0)anintegernumberandisuniquewithinthelistofdocumentrevisionsforasingledocument.InArangoDB>=3.1the_revstringsareinfacttimestamps.TheyusethelocalclockoftheDBserverthatactuallywritesthedocumentandhavemillisecondaccuracy.Actually,a"HybridLogicalClock"isused(forthisconceptseethispaper).

Withinonesharditisguaranteedthattwodifferentdocumentrevisionshaveadifferent_revstring,eveniftheyarewritteninthesamemillisecond,andthatthesestampsareascending.


66

http://www.cse.buffalo.edu/tech-reports/2014-04.pdf

Notehoweverthatdifferentserversinyourclustermighthaveaclockskew,andthereforebetweendifferentshardsorevenbetweendifferentcollectionsthetimestampsarenotguaranteedtobecomparable.

TheHybridLogicalClockfeaturedoesonethingtoaddressthisissue:WheneveramessageissentfromsomeserverAinyourclustertoanotheroneB,itisensuredthatanytimestamptakenonBafterthemessagehasarrivedisgreaterthananytimestamptakenonAbeforethemessagewassent.Thisensuresthatifthereissome"causality"betweeneventsondifferentservers,timestampsincreasefromcausetoeffect.Adirectconsequenceofthisisthatsometimesaserverhastotaketimestampsthatseemtocomefromthefutureofitsownclock.Itwillhoweverstillproduceeverincreasingtimestamps.Iftheclockskewissmall,thenyourtimestampswillrelativelyaccuratelydescribethetimewhenthedocumentrevisionwasactuallywritten.

ArangoDBuses64bitunsignedintegervaluestomaintaindocumentrevisionsinternally.Atthisstageweintentionallydonotdocumenttheexactformatoftherevisionvalues.Whenreturningdocumentrevisionstoclients,ArangoDBwillputthemintoastringtoensuretherevisionisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattherevisionreturnedbyArangoDBasanopaquestringwhentheystoreoruseitlocally.ThiswillallowArangoDBtochangetheformatofrevisionslaterifthisshouldberequired(ashashappenedwith3.1withtheHybridLogicalClock).Clientscanuserevisionstoperformsimpleequality/non-equalitycomparisons(e.g.tocheckwhetheradocumenthaschangedornot),buttheyshouldnotuserevisionidstoperformgreater/lessthancomparisonswiththemtocheckifadocumentrevisionisolderthanoneanother,evenifthismightworkforsomecases.

Documentrevisionscanbeusedtoconditionallyquery,update,replaceordeletedocumentsinthedatabase.Inordertofindaparticularrevisionofadocument,youneedthedocumenthandleorkey,andthedocumentrevision.

MultipleDocumentsinasingleCommand

BeginningwithArangoDB3.0thebasicdocumentAPIhasbeenextendedtohandlenotonlysingledocumentsbutmultipledocumentsinasinglecommand.Thisiscrucialforperformance,inparticularintheclustersituation,inwhichasinglerequestcaninvolvemultiplenetworkhopswithinthecluster.Anotheradvantageisthatitreducestheoverheadofindividualnetworkroundtripsbetweentheclientandtheserver.ThegeneralideatoperformmultipledocumentoperationsinasinglecommandistouseJSONarraysofobjectsintheplaceofasingledocument.Asaconsequence,documentkeys,handlesandrevisionsforpreconditionshavetobesuppliedembeddedintheindividualdocumentsgiven.Multipledocumentoperationsarerestrictedtoasingledocumentoredgecollection.SeetheAPIdescriptionsforcollectionobjectsfordetails.NotethattheAPIfordatabaseobjectsdonotoffertheseoperations.


67

CollectionMethods

All

collection.all()

Fetchesalldocumentsfromacollectionandreturnsacursor.YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Examples

UsetoArraytogetalldocumentsatonce:

arangosh>db.five.save({name:"one"});

arangosh>db.five.save({name:"two"});

arangosh>db.five.save({name:"three"});

arangosh>db.five.save({name:"four"});

arangosh>db.five.save({name:"five"});

arangosh>db.five.all().toArray();

showexecutionresultsUselimittorestrictthedocuments:

arangosh>db.five.save({name:"one"});

arangosh>db.five.save({name:"two"});

arangosh>db.five.save({name:"three"});

arangosh>db.five.save({name:"four"});

arangosh>db.five.save({name:"five"});

arangosh>db.five.all().limit(2).toArray();


Querybyexamplecollection.byExample(example)

Fetchesalldocumentsfromacollectionthatmatchthespecifiedexampleandreturnsacursor.

YoucanusetoArray,next,orhasNexttoaccesstheresult.Theresultcanbelimitedusingtheskipandlimitoperator.

Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.Ifyouuse

{"a":{"c":1}}

asexample,thenyouwillfindalldocuments,suchthattheattributeacontainsadocumentoftheform{c:1}.Forexamplethedocument

{"a":{"c":1},"b":1}

willmatch,butthedocument

{"a":{"c":1,"b":1}}

willnot.

However,ifyouuse

{"a.c":1}

CollectionMethods

68

thenyouwillfindalldocuments,whichcontainasub-documentinathathasanattributecofvalue1.Boththefollowingdocuments

{"a":{"c":1},"b":1}

and

{"a":{"c":1,"b":1}}

willmatch.

collection.byExample(path1,value1,...)

Asalternativeyoucansupplyanarrayofpathsandvalues.

Examples


arangosh>db.users.save({name:"Gerhard"});

arangosh>db.users.save({name:"Helmut"});

arangosh>db.users.save({name:"Angela"});

arangosh>db.users.all().toArray();

arangosh>db.users.byExample({"_id":"users/20"}).toArray();

arangosh>db.users.byExample({"name":"Gerhard"}).toArray();

arangosh>db.users.byExample({"name":"Helmut","_id":"users/15"}).toArray();

showexecutionresultsUsenexttoloopoveralldocuments:

arangosh>db.users.save({name:"Gerhard"});

arangosh>db.users.save({name:"Helmut"});

arangosh>db.users.save({name:"Angela"});

arangosh>vara=db.users.byExample({"name":"Angela"});

arangosh>while(a.hasNext())print(a.next());


FirstExample

collection.firstExample(example)

Returnssomedocumentofacollectionthatmatchesthespecifiedexample.Ifnosuchdocumentexists,nullwillbereturned.Theexamplehastobespecifiedaspathsandvalues.SeebyExamplefordetails.

collection.firstExample(path1,value1,...)

Asalternativeyoucansupplyanarrayofpathsandvalues.

Examples

arangosh>db.users.firstExample("name","Angela");


Range

collection.range(attribute,left,right)

CollectionMethods

69

Returnsalldocumentsfromacollectionsuchthattheattributeisgreaterorequalthanleftandstrictlylessthanright.


Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.

Note:therangesimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionwithinaspecificrangeistouseanAQLqueryasfollows:

FORdocIN@@collection

FILTERdoc.value>=@left&&doc.value<@right

LIMIT@skip,@limit

RETURNdoc

Examples


arangosh>db.old.ensureIndex({type:"skiplist",fields:["age"]});

arangosh>db.old.save({age:15});



arangosh>db.old.range("age",10,30).toArray();


Closedrange

collection.closedRange(attribute,left,right)

Returnsalldocumentsofacollectionsuchthattheattributeisgreaterorequalthanleftandlessorequalthanright.


Anattributenameoftheforma.bisinterpretedasattributepath,notasattribute.

Note:theclosedRangesimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionwithinaspecificrangeistouseanAQLqueryasfollows:


FILTERdoc.value>=@left&&doc.value<=@right

LIMIT@skip,@limit

RETURNdoc

Examples


arangosh>db.old.ensureIndex({type:"skiplist",fields:["age"]});




arangosh>db.old.closedRange("age",10,30).toArray();


Any

collection.any()

Returnsarandomdocumentfromthecollectionornullifnoneexists.

Note:thismethodisexpensivewhenusingtheRocksDBstorageengine.

CollectionMethods

70

Count

collection.count()

Returnsthenumberoflivingdocumentsinthecollection.

Examples

arangosh>db.users.count();

0

toArray

collection.toArray()

Convertsthecollectionintoanarrayofdocuments.NeverusethiscallinaproductionenvironmentasitwillbasicallycreateacopyofyourcollectioninRAMwhichwilluseresourcesdependingonthenumberandsizeofthedocumentsinyourcollecion.

Documentcollection.document(object)

Thedocumentmethodfindsadocumentgivenanobjectobjectcontainingthe_idor_keyattribute.Themethodreturnsthedocumentifitcanbefound.Ifbothattributesaregiven,the_idtakesprecedence,itisanerror,ifthecollectionpartofthe_iddoesnotmatchthecollection.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.Anerrorisalsothrownifnodocumentexistswiththegiven_idor_keyvalue.

Pleasenotethatifthemethodisexecutedonthearangodserver(e.g.frominsideaFoxxapplication),animmutabledocumentobjectwillbereturnedforperformancereasons.Itisnotpossibletochangeattributesofthisimmutableobject.Toupdateorpatchthereturneddocument,itneedstobecloned/copiedintoaregularJavaScriptobjectfirst.Thisisnotnecessaryifthedocumentmethodiscalledfromoutofarangoshorfromanyotherclient.

collection.document(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.Norevisioncanbespecifiedinthiscase.

collection.document(document-key)

Asbefore.Insteadofobjectadocument-keycanbepassedasfirstargument.

collection.document(array)

Thisvariantallowstoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifdocumentwouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.

Examples

Returnsthedocumentforadocument-handle:

arangosh>db.example.document("example/2873916");

showexecutionresultsReturnsthedocumentforadocument-key:

arangosh>db.example.document("2873916");

showexecutionresultsReturnsthedocumentforanobject:

arangosh>db.example.document({_id:"example/2873916"});

CollectionMethods

71

showexecutionresultsReturnsthedocumentforanarrayoftwokeys:

arangosh>db.example.document(["2873916","2873917"]);

showexecutionresultsAnerrorisraisedifthedocumentisunknown:


[ArangoError1202:documentnotfound]

Anerrorisraisedifthehandleisinvalid:

arangosh>db.example.document("");

[ArangoError1205:illegaldocumenthandle]

Changesin3.0from2.8:

documentcannowquerymultipledocumentswithonecall.

Exists

checkswhetheradocumentexistscollection.exists(object)

Theexistsmethoddetermineswhetheradocumentexistsgivenanobjectobjectcontainingthe_idor_keyattribute.Ifbothattributesaregiven,the_idtakesprecedence,itisanerror,ifthecollectionpartofthe_iddoesnotmatchthecollection.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.

Insteadofreturningthefounddocumentoranerror,thismethodwillonlyreturnanobjectwiththeattributes_id,_keyand_rev,orfalseifnodocumentwiththegiven_idor_keyexists.Itcanthusbeusedforeasyexistencechecks.

Thismethodwillthrowanerrorifusedimproperly,e.g.whencalledwithanon-documenthandle,anon-document,orwhenacross-collectionrequestisperformed.

collection.exists(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.

collection.exists(document-key)

Asbefore.Insteadofobjectadocument-keycanbepassedasfirstargument.

collection.exists(array)

Thisvariantallowstoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifexistswouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,theoperationstopsimmediatelyreturningonlyanerrorobject.


Inthecaseofarevisionmismatchexistsnowthrowsanerrorinsteadofsimplyreturningfalse.Thisistomakeitpossibletotellthedifferencebetweenarevisionmismatchandanon-existingdocument.

existscannowquerymultipledocumentswithonecall.

LookupByKeyscollection.documents(keys)

CollectionMethods

72

Looksupthedocumentsinthespecifiedcollectionusingthearrayofkeysprovided.Alldocumentsforwhichamatchingkeywasspecifiedinthekeysarrayandthatexistinthecollectionwillbereturned.Keysforwhichnodocumentcanbefoundintheunderlyingcollectionareignored,andnoexceptionwillbethrownforthem.

Thismethodisdeprecatedinfavourofthearrayvariantofdocument.

Examples

arangosh>keys=[];

arangosh>for(vari=0;i<10;++i){

........>db.example.insert({_key:"test"+i,value:i});

........>keys.push("test"+i);

........>}

arangosh>db.example.documents(keys);


Insert

collection.insert(data)

Createsanewdocumentinthecollectionfromthegivendata.Thedatamustbeanobject.Theattributes_idand_revareignoredandareautomaticallygenerated.Auniquevaluefortheattribute_keywillbeautomaticallygeneratedifnotspecified.Ifspecified,theremustnotbeadocumentwiththegiven_keyinthecollection.

Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleofthenewlycreateddocument,theattribute_keythedocumentkeyandtheattribute_revcontainsthedocumentrevision.

collection.insert(data,options)

Createsanewdocumentinthecollectionfromthegivendataasabove.Theoptionaloptionsparametermustbeanobjectandcanbeusedtospecifythefollowingoptions:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.silent:Ifthisflagissettotrue,themethoddoesnotreturnanyoutput.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.

Note:sinceArangoDB2.2,insertisanaliasforsave.

collection.insert(array)

collection.insert(array,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofarguments.Thebehaviorisexactlyasifinsertwouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.


TheoptionssilentandreturnNewarenew.Themethodcannowinsertmultipledocumentswithonecall.

Examples

arangosh>db.example.insert({Hello:"World"});

arangosh>db.example.insert({Hello:"World"},{waitForSync:true});


arangosh>db.example.insert([{Hello:"World"},{Hello:"there"}])

CollectionMethods

73

arangosh>db.example.insert([{Hello:"World"},{}],{waitForSync:true});


Replace

collection.replace(selector,data)

Replacesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenreplacedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowreplaced)document.

Iftheselectorcontainsa_revattribute,themethodfirstchecksthatthespecifiedrevisionisthecurrentrevisionofthatdocument.Ifnot,thereisaconflict,andanerroristhrown.

collection.replace(selector,data,options)

Asbefore,butoptionsmustbeanobjectthatcancontainthefollowingbooleanattributes:

waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

collection.replace(document-handle,data)

collection.replace(document-handle,data,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisionpreconditionistested.

collection.replace(document-key,data)

collection.replace(document-key,data,options)

Asbefore.Insteadofselectoradocument-keycanbepassedasfirstargument.Norevisionpreconditionistested.

collection.replace(selectorarray,dataarray)

collection.replace(selectorarray,dataarray,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofselector/datapairs.Thetwoarraysgivenasselectorarrayanddataarraymusthavethesamelength.Thebehaviorisexactlyasifreplacewouldhavebeencalledonallrespectivemembersofthetwoarraysandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Examples

Createandupdateadocument:

arangosh>a1=db.example.insert({a:1});

arangosh>a2=db.example.replace(a1,{a:2});


showexecutionresultsUseadocumenthandle:


CollectionMethods

74

arangosh>a2=db.example.replace("example/3903044",{a:2});



Theoptionssilent,returnNewandreturnOldarenew.Themethodcannowreplacemultipledocumentswithonecall.

Update

collection.update(selector,data)

Updatesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenpatchedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowupdated)document.


collection.update(selector,data,options)


waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.keepNull:TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.mergeObjects:Controlswhetherobjects(notarrays)willbemergedifpresentinboththeexistingandthepatchdocument.Ifsettofalse,thevalueinthepatchdocumentwilloverwritetheexistingdocument'svalue.Ifsettotrue,objectswillbemerged.Thedefaultistrue.

collection.update(document-handle,data)

collection.update(document-handle,data,options)


collection.update(document-key,data)

collection.update(document-key,data,options)

Asbefore.Insteadofselectoradocument-keycanbepassedasfirstargument.Norevisionpreconditionistested.

collection.update(selectorarray,dataarray)

collection.update(selectorarray,dataarray,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofselector/datapairs.Thetwoarraysgivenasselectorarrayanddataarraymusthavethesamelength.Thebehaviorisexactlyasifupdatewouldhavebeencalledonallrespectivemembersofthetwoarraysandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Examples


CollectionMethods

75

arangosh>a1=db.example.insert({"a":1});

arangosh>a2=db.example.update(a1,{"b":2,"c":3});

arangosh>a3=db.example.update(a1,{"d":4});

arangosh>a4=db.example.update(a2,{"e":5,"f":6});

arangosh>db.example.document(a4);

arangosh>a5=db.example.update(a4,{"a":1,c:9,e:42});


showexecutionresultsUseadocumenthandle:

arangosh>a1=db.example.insert({"a":1});

arangosh>a2=db.example.update("example/18612115",{"x":1,"y":2});

showexecutionresultsUsethekeepNullparametertoremoveattributeswithnullvalues:

arangosh>db.example.insert({"a":1});

arangosh>db.example.update("example/19988371",

........>{"b":null,"c":null,"d":3});


arangosh>db.example.update("example/19988371",{"a":null},false,false);


arangosh>db.example.update("example/19988371",

........>{"b":null,"c":null,"d":null},false,false);


showexecutionresultsPatchingarrayvalues:

arangosh>db.example.insert({"a":{"one":1,"two":2,"three":3},

........>"b":{}});

arangosh>db.example.update("example/20774803",{"a":{"four":4},

........>"b":{"b1":1}});


arangosh>db.example.update("example/20774803",{"a":{"one":null},

........>"b":null},

........>false,false);




Theoptionssilent,returnNewandreturnOldarenew.Themethodcannowupdatemultipledocumentswithonecall.

Remove

collection.remove(selector)

Removesadocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idor_keyattribute.Theremustbeadocumentwiththat_idor_keyinthecurrentcollection.Thisdocumentisthenremoved.

Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleoftheremoveddocument,theattribute_revcontainsthedocumentrevisionoftheremoveddocument.

CollectionMethods

76


collection.remove(selector,options)


waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

collection.remove(document-handle)

collection.remove(document-handle,options)

Asbefore.Insteadofselectoradocument-handlecanbepassedasfirstargument.Norevisioncheckisperformed.

collection.remove(document-key)

collection.remove(document-handle,options)


collection.remove(selectorarray)

collection.remove(selectorarray,options)

Thesetwovariantsallowtoperformtheoperationonawholearrayofselectors.Thebehaviorisexactlyasifremovewouldhavebeencalledonallmembersofthearrayseparatelyandallresultsarereturnedinanarray.Ifanerroroccurswithanyofthedocuments,noexceptionisrisen!Insteadofadocumentanerrorobjectisreturnedintheresultarray.Theoptionsbehaveexactlyasbefore.

Examples

Removeadocument:



arangosh>db.example.remove(a1);


showexecutionresultsRemoveadocumentwithaconflict:



arangosh>db.example.remove(a1);

arangosh>db.example.remove(a1,true);




Themethodnowreturnsnotonlytruebutinformationabouttheremoveddocument(s).TheoptionssilentandreturnOldarenew.Themethodcannowremovemultipledocumentswithonecall.

RemoveByKeys

collection.removeByKeys(keys)

CollectionMethods

77

Looksupthedocumentsinthespecifiedcollectionusingthearrayofkeysprovided,andremovesalldocumentsfromthecollectionwhosekeysarecontainedinthekeysarray.Keysforwhichnodocumentcanbefoundintheunderlyingcollectionareignored,andnoexceptionwillbethrownforthem.

Themethodwillreturnanobjectcontainingthenumberofremoveddocumentsintheremovedsub-attribute,andthenumberofnot-removed/ignoreddocumentsintheignoredsub-attribute.

Thismethodisdeprecatedinfavourofthearrayvariantofremove.

Examples

arangosh>keys=[];

arangosh>for(vari=0;i<10;++i){

........>db.example.insert({_key:"test"+i,value:i});

........>keys.push("test"+i);

........>}

arangosh>db.example.removeByKeys(keys);


RemoveByExamplecollection.removeByExample(example)

Removesalldocumentsmatchinganexample.

collection.removeByExample(document,waitForSync)

TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentdeletionoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.

collection.removeByExample(document,waitForSync,limit)

Theoptionallimitparametercanbeusedtorestrictthenumberofremovalstothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsareremoved.

Examples

arangosh>db.example.removeByExample({Hello:"world"});

1

ReplaceByExample

collection.replaceByExample(example,newValue)

Replacesalldocumentsmatchinganexamplewithanewdocumentbody.TheentiredocumentbodyofeachdocumentmatchingtheexamplewillbereplacedwithnewValue.Thedocumentmeta-attributes_id,_keyand_revwillnotbereplaced.

collection.replaceByExample(document,newValue,waitForSync)

TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentreplacementoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.

collection.replaceByExample(document,newValue,waitForSync,limit)

Theoptionallimitparametercanbeusedtorestrictthenumberofreplacementstothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsarereplaced.

CollectionMethods

78

Examples

arangosh>db.example.save({Hello:"world"});

arangosh>db.example.replaceByExample({Hello:"world"},{Hello:"mars"},false,5);


UpdateByExamplecollection.updateByExample(example,newValue)

Partiallyupdatesalldocumentsmatchinganexamplewithanewdocumentbody.SpecificattributesinthedocumentbodyofeachdocumentmatchingtheexamplewillbeupdatedwiththevaluesfromnewValue.Thedocumentmeta-attributes_id,_keyand_revcannotbeupdated.

Partialupdatecouldalsobeusedtoappendnewfields,iftherewerenooldfieldwithsamename.

collection.updateByExample(document,newValue,keepNull,waitForSync)

TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.

TheoptionalwaitForSyncparametercanbeusedtoforcesynchronizationofthedocumentreplacementoperationtodiskevenincasethatthewaitForSyncflaghadbeendisabledfortheentirecollection.Thus,thewaitForSyncparametercanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.

collection.updateByExample(document,newValue,keepNull,waitForSync,limit)

Theoptionallimitparametercanbeusedtorestrictthenumberofupdatestothespecifiedvalue.Iflimitisspecifiedbutlessthanthenumberofdocumentsinthecollection,itisundefinedwhichdocumentsareupdated.

collection.updateByExample(document,newValue,options)

Usingthisvariant,theoptionsfortheoperationcanbepassedusinganobjectwiththefollowingsub-attributes:

keepNullwaitForSynclimitmergeObjects

Examples

arangosh>db.example.save({Hello:"world",foo:"bar"});

arangosh>db.example.updateByExample({Hello:"world"},{Hello:"foo",World:"bar"},

false);

arangosh>db.example.byExample({Hello:"foo"}).toArray()


Collectiontypecollection.type()

Returnsthetypeofacollection.Possiblevaluesare:

2:documentcollection3:edgecollection

GettheVersionofArangoDBdb._version()

CollectionMethods

79

Returnstheserverversionstring.Notethatthisisnottheversionofthedatabase.

Examples

arangosh>require("@arangodb").db._version();

3.3.10

Edges

Edgesarenormaldocumentsthatalwayscontaina_fromanda_toattribute.Therefore,youcanusethedocumentmethodstooperateonedges.Thefollowingmethods,however,arespecifictoedges.

edge-collection.edges(vertex)

Theedgesoperatorfindsalledgesstartingfrom(outbound)orendingin(inbound)vertex.

edge-collection.edges(vertices)

Theedgesoperatorfindsalledgesstartingfrom(outbound)orendingin(inbound)adocumentfromvertices,whichmustbealistofdocumentsordocumenthandles.

arangosh>db._create("vertex");

arangosh>db._createEdgeCollection("relation");

arangosh>varmyGraph={};

arangosh>myGraph.v1=db.vertex.insert({name:"vertex1"});


arangosh>myGraph.e1=db.relation.insert(myGraph.v1,myGraph.v2,

........>{label:"knows"});

arangosh>db._document(myGraph.e1);

arangosh>db.relation.edges(myGraph.e1._id);

showexecutionresultsedge-collection.inEdges(vertex)

Theedgesoperatorfindsalledgesendingin(inbound)vertex.

edge-collection.inEdges(vertices)

Theedgesoperatorfindsalledgesendingin(inbound)adocumentfromvertices,whichmustalistofdocumentsordocumenthandles.

Examples






........>{label:"knows"});


arangosh>db.relation.inEdges(myGraph.v1._id);

arangosh>db.relation.inEdges(myGraph.v2._id);

showexecutionresultsedge-collection.outEdges(vertex)

Theedgesoperatorfindsalledgesstartingfrom(outbound)vertices.

edge-collection.outEdges(vertices)

Theedgesoperatorfindsalledgesstartingfrom(outbound)adocumentfromvertices,whichmustalistofdocumentsordocumenthandles.

Examples

CollectionMethods

80






........>{label:"knows"});


arangosh>db.relation.outEdges(myGraph.v1._id);

arangosh>db.relation.outEdges(myGraph.v2._id);


Misc

collection.iterate(iterator,options)

Iteratesoversomeelementsofthecollectionandapplythefunctioniteratortotheelements.Thefunctionwillbecalledwiththedocumentasfirstargumentandthecurrentnumber(startingwith0)assecondargument.

optionsmustbeanobjectwiththefollowingattributes:

limit(optional,defaultnone):useatmostlimitdocuments.

probability(optional,defaultall):anumberbetween0and1.Documentsarechosenwiththisprobability.

Examples

arangosh>for(i=-90;i<=90;i+=10){

........>for(j=-180;j<=180;j+=10){

........>db.example.save({name:"Name/"+i+"/"+j,

........>home:[i,j],

........>work:[-i,-j]});

........>}

........>}

........>

arangosh>db.example.ensureIndex({type:"geo",fields:["home"]});

arangosh>items=db.example.getIndexes().map(function(x){returnx.id;});

........>db.example.index(items[1]);


CollectionMethods

81

DatabaseMethods

Document

db._document(object)

The_documentmethodfindsadocumentgivenanobjectobjectcontainingthe_idattribute.Themethodreturnsthedocumentifitcanbefound.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.Anerrorisalsothrownifnodocumentexistswiththegiven_id.

Pleasenotethatifthemethodisexecutedonthearangodserver(e.g.frominsideaFoxxapplication),animmutabledocumentobjectwillbereturnedforperformancereasons.Itisnotpossibletochangeattributesofthisimmutableobject.Toupdateorpatchthereturneddocument,itneedstobecloned/copiedintoaregularJavaScriptobjectfirst.Thisisnotnecessaryifthe_documentmethodiscalledfromoutofarangoshorfromanyotherclient.

db._document(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.Norevisioncanbespecifiedinthiscase.

Examples

Returnsthedocument:

arangosh>db._document("example/12345");


Existsdb._exists(object)

The_existsmethoddetermineswhetheradocumentexistsgivenanobjectobjectcontainingthe_idattribute.

Anerroristhrownif_revisspecifiedbutthedocumentfoundhasadifferentrevisionalready.

Insteadofreturningthefounddocumentoranerror,thismethodwillonlyreturnanobjectwiththeattributes_id,_keyand_rev,orfalseifnodocumentwiththegiven_idor_keyexists.Itcanthusbeusedforeasyexistencechecks.

Thismethodwillthrowanerrorifusedimproperly,e.g.whencalledwithanon-documenthandle,anon-document,orwhenacross-collectionrequestisperformed.

db._exists(document-handle)

Asbefore.Insteadofobjectadocument-handlecanbepassedasfirstargument.


Inthecaseofarevisionmismatch_existsnowthrowsanerrorinsteadofsimplyreturningfalse.Thisistomakeitpossibletotellthedifferencebetweenarevisionmismatchandanon-existingdocument.

Replace

db._replace(selector,data)

Replacesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenreplacedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowreplaced)document.

DatabaseMethods

82


collection.replace(selector,data,options)


waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

db._replace(document-handle,data)

db._replace(document-handle,data,options)


Examples

Createandreplaceadocument:


arangosh>a2=db._replace(a1,{a:2});




Theoptionssilent,returnNewandreturnOldarenew.

Update

db._update(selector,data)

Updatesanexistingdocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenpatchedwiththedatagivenassecondargument.Anyattribute_id,_keyor_revindataisignored.

Themethodreturnsadocumentwiththeattributes_id,_key,_revand_oldRev.Theattribute_idcontainsthedocumenthandleoftheupdateddocument,theattribute_revcontainsthedocumentrevisionoftheupdateddocument,theattribute_oldRevcontainstherevisionoftheold(nowupdated)document.


db._update(selector,data,options)


waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnNew:Ifthisflagissettotrue,thecompletenewdocumentisreturnedintheoutputundertheattributenew.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.

DatabaseMethods

83

silent:Ifthisflagissettotrue,nooutputisreturned.keepNull:TheoptionalkeepNullparametercanbeusedtomodifythebehaviorwhenhandlingnullvalues.Normally,nullvaluesarestoredinthedatabase.BysettingthekeepNullparametertofalse,thisbehaviorcanbechangedsothatallattributesindatawithnullvalueswillberemovedfromthetargetdocument.mergeObjects:Controlswhetherobjects(notarrays)willbemergedifpresentinboththeexistingandthepatchdocument.Ifsettofalse,thevalueinthepatchdocumentwilloverwritetheexistingdocument'svalue.Ifsettotrue,objectswillbemerged.Thedefaultistrue.

db._update(document-handle,data)

db._update(document-handle,data,options)


Examples



arangosh>a2=db._update(a1,{b:2});

arangosh>a3=db._update(a1,{c:3});



Theoptionssilent,returnNewandreturnOldarenew.

Removedb._remove(selector)

Removesadocumentdescribedbytheselector,whichmustbeanobjectcontainingthe_idattribute.Theremustbeadocumentwiththat_idinthecurrentdatabase.Thisdocumentisthenremoved.

Themethodreturnsadocumentwiththeattributes_id,_keyand_rev.Theattribute_idcontainsthedocumenthandleoftheremoveddocument,theattribute_revcontainsthedocumentrevisionoftheremovedeocument.


db._remove(selector,options)


waitForSync:OnecanforcesynchronizationofthedocumentcreationoperationtodiskevenincasethatthewaitForSyncflagisbeendisabledfortheentirecollection.Thus,thewaitForSyncoptioncanbeusedtoforcesynchronizationofjustspecificoperations.Tousethis,setthewaitForSyncparametertotrue.IfthewaitForSyncparameterisnotspecifiedorsettofalse,thenthecollection'sdefaultwaitForSyncbehaviorisapplied.ThewaitForSyncparametercannotbeusedtodisablesynchronizationforcollectionsthathaveadefaultwaitForSyncvalueoftrue.overwrite:Ifthisflagissettotrue,a_revattributeintheselectorisignored.returnOld:Ifthisflagissettotrue,thecompletepreviousrevisionofthedocumentisreturnedintheoutputundertheattributeold.silent:Ifthisflagissettotrue,nooutputisreturned.

db._remove(document-handle)

db._remove(document-handle,options)


Examples

Removeadocument:


DatabaseMethods

84

arangosh>db._remove(a1);


arangosh>db._remove(a1,{overwrite:true});

showexecutionresultsRemovethedocumentintherevisiona1withaconflict:




arangosh>db._remove(a1,{overwrite:true});

arangosh>db._document(a1);

showexecutionresultsRemoveadocumentusingnewsignature:

arangosh>db.example.insert({_key:"11265325374",a:1});

arangosh>db.example.remove("example/11265325374",

........>{overwrite:true,waitForSync:false})



Themethodnowreturnsnotonlytruebutinformationabouttheremoveddocument(s).TheoptionssilentandreturnOldarenew.

DatabaseMethods

85

Graphs,Vertices&EdgesGraphs,vertices&edgesaredefinedintheGraphschapterindetails.

Relatedblogposts:

Graphsindatamodeling-istheemperornaked?IndexFreeAdjacencyorHybridIndexesforGraphDatabases

Graphs,Vertices&Edges

86

https://medium.com/@neunhoef/graphs-in-data-modeling-is-the-emperor-naked-2e65e2744413#.x0a5z66ji

https://www.arangodb.com/2016/04/index-free-adjacency-hybrid-indexes-graph-databases/

NamingConventionsinArangoDBThefollowingnamingconventionsshouldbefollowedbyuserswhencreatingdatabases,collectionsanddocumentsinArangoDB.

NamingConventions

87

DatabaseNamesArangoDBwillalwaysstartupwithadefaultdatabase,named_system.UserscancreateadditionaldatabasesinArangoDB,providedthedatabasenamesconformtothefollowingconstraints:

Databasenamesmustonlyconsistofthelettersatoz(bothloweranduppercaseallowed),thenumbers0to9,andtheunderscore(_)ordash(-)symbolsThisalsomeansthatanynon-ASCIIdatabasenamesarenotallowedDatabasenamesmustalwaysstartwithaletter.Databasenamesstartingwithanunderscoreareconsideredtobesystemdatabases,andusersshouldnotcreateordeletethoseThemaximumallowedlengthofadatabasenameis64bytesDatabasenamesarecase-sensitive

DatabaseNames

88

CollectionNamesUserscanpicknamesfortheircollectionsasdesired,providedthefollowingnamingconstraintsarenotviolated:

Collectionnamesmustonlyconsistofthelettersatoz(bothinloweranduppercase),thenumbers0to9,andtheunderscore(_)ordash(-)symbols.Thisalsomeansthatanynon-ASCIIcollectionnamesarenotallowedUser-definedcollectionnamesmustalwaysstartwithaletter.Systemcollectionnamesmuststartwithanunderscore.AllcollectionnamesstartingwithanunderscoreareconsideredtobesystemcollectionsthatareforArangoDB'sinternaluseonly.SystemcollectionnamesshouldnotbeusedbyendusersfortheirowncollectionsThemaximumallowedlengthofacollectionnameis64bytesCollectionnamesarecase-sensitive

CollectionNames

89

DocumentKeysUserscandefinetheirownkeysfordocumentstheysave.Thedocumentkeywillbesavedalongwithadocumentinthe_keyattribute.Userscanpickkeyvaluesasrequired,providedthatthevaluesconformtothefollowingrestrictions:

Thekeymustbeastringvalue.Numerickeysarenotallowed,butanynumericvaluecanbeputintoastringandcanthenbeusedasdocumentkey.Thekeymustbeatleast1byteandatmost254byteslong.Emptykeysaredisallowedwhenspecified(thoughitmaybevalidtocompletelyomitthe_keyattributefromadocument)Itmustconsistofthelettersa-z(loweroruppercase),thedigits0-9oranyofthefollowingpunctuationcharacters:_-:.@()+,=;$!*'%Anyothercharacters,especiallymulti-byteUTF-8sequences,whitespaceorpunctuationcharacterscannotbeusedinsidekeyvaluesThekeymustbeuniquewithinthecollectionitisused

Keysarecase-sensitive,i.e.myKeyandMyKEYareconsideredtobedifferentkeys.

Specifyingadocumentkeyisoptionalwhencreatingnewdocuments.Ifnodocumentkeyisspecifiedbytheuser,ArangoDBwillcreatethedocumentkeyitselfaseachdocumentisrequiredtohaveakey.

Therearenoguaranteesabouttheformatandpatternofauto-generateddocumentkeysotherthantheaboverestrictions.Clientsshouldthereforetreatauto-generateddocumentkeysasopaquevaluesandnotrelyontheirformat.

Thecurrentformatforgeneratedkeysisastringcontainingnumericdigits.Thenumericvaluesreflectchronologicaltimeinthesensethat_keyvaluesgeneratedlaterwillcontainhighernumbersthan_keyvaluesgeneratedearlier.Buttheexactvaluethatwillbegeneratedbytheserverisnotpredictable.Notethatifyousortonthe_keyattribute,stringcomparisonwillbeused,whichmeans"100"islessthan"99"etc.

DocumentKeys

90

AttributeNamesUserscanpickattributenamesfordocumentattributesasdesired,providedthefollowingattributenamingconstraintsarenotviolated:

AttributenamesstartingwithanunderscoreareconsideredtobesystemattributesforArangoDB'sinternaluse.SuchattributenamesarealreadyusedbyArangoDBforspecialpurposes:

_idisusedtocontainadocument'shandle_keyisusedtocontainadocument'suser-definedkey_revisusedtocontainthedocument'srevisionnumberInedgecollections,the

_from_to

attributesareusedtoreferenceotherdocuments.

Moresystemattributesmaybeaddedinthefuturewithoutfurthernoticesoendusersshouldtrytoavoidusingtheirownattributenamesstartingwithunderscores.

Theoretically,attributenamescanincludepunctuationandspecialcharactersasdesired,providedthenameisavalidUTF-8string.Formaximumportability,specialcharactersshouldbeavoidedthough.Forexample,attributenamesmaycontainthedotsymbol,butthedothasaspecialmeaninginJavaScriptandalsoinAQL,sowhenusingsuchattributenamesinoneoftheselanguages,theattributenameneedstobequotedbytheenduser.Overallitmightbebettertouseattributenameswhichdon'trequireanyquoting/escapinginalllanguagesused.Thisincludeslanguagesusedbytheclient(e.g.Ruby,PHP)iftheattributesaremappedtoobjectmembersthere.

Attributenamesstartingwithanat-mark(@)willneedtobeenclosedinbacktickswhenusedinanAQLquerytotellthemapartfrombindvariables.Thereforewedonotencouragetheuseofattributesstartingwithat-marks,thoughtheywillworkwhenusedproperly.ArangoDBdoesnotenforcealengthlimitforattributenames.However,longattributenamesmayusemorememoryinresultsetsetc.Thereforetheuseoflongattributenamesisdiscouraged.Attributenamesarecase-sensitive.Attributeswithemptynames(anemptystring)aredisallowed.

AttributeNames

91

HandlingIndexesThisisanintroductiontoArangoDB'sinterfaceforindexesingeneral.Therearespecialsectionsfor

IndexBasics:IntroductiontoallindextypesWhichindextousewhen:IndextypeandoptionsadviserIndexUtilization:HowArangoDBusesindexesWorkingwithIndexes:Howtohandleindexesprogrammaticallyusingthedbobject

HashIndexesSkiplistsPersistentIndexesFulltextIndexesGeo-spatialIndexesVertex-centricIndexes

Indexing

92

Indexbasics

Indexesallowfastaccesstodocuments,providedtheindexedattribute(s)areusedinaquery.WhileArangoDBautomaticallyindexessomesystemattributes,usersarefreetocreateextraindexesonnon-systemattributesofdocuments.

User-definedindexescanbecreatedoncollectionlevel.Mostuser-definedindexescanbecreatedbyspecifyingthenamesoftheindexattributes.Someindextypesallowindexingjustoneattribute(e.g.fulltextindex)whereasotherindextypesallowindexingmultipleattributesatthesametime.

Thesystemattributes_id,_key,_fromand_toareautomaticallyindexedbyArangoDB,withouttheuserbeingrequiredtocreateextraindexesforthem._idand_keyarecoveredbyacollection'sprimarykey,and_fromand_toarecoveredbyanedgecollection'sedgeindexautomatically.

Usingthesystemattribute_idinuser-definedindexesisnotpossible,butindexing_key,_rev,_from,and_tois.

ArangoDBprovidesthefollowingindextypes:

PrimaryIndex

Foreachcollectiontherewillalwaysbeaprimaryindexwhichisahashindexforthedocumentkeys(_keyattribute)ofalldocumentsinthecollection.Theprimaryindexallowsquickselectionofdocumentsinthecollectionusingeitherthe_keyor_idattributes.ItwillbeusedfromwithinAQLqueriesautomaticallywhenperformingequalitylookupson_keyor_id.

Therearealsodedicatedfunctionstofindadocumentgivenits_keyor_idthatwillalwaysmakeuseoftheprimaryindex:

db.collection.document("<document-key>");

db._document("<document-id>");

Astheprimaryindexisanunsortedhashindex,itcannotbeusedfornon-equalityrangequeriesorforsorting.

Theprimaryindexofacollectioncannotbedroppedorchanged,andthereisnomechanismtocreateuser-definedprimaryindexes.

EdgeIndex

Everyedgecollectionalsohasanautomaticallycreatededgeindex.Theedgeindexprovidesquickaccesstodocumentsbyeithertheir_fromor_toattributes.Itcanthereforebeusedtoquicklyfindconnectionsbetweenvertexdocumentsandisinvokedwhentheconnectingedgesofavertexarequeried.

EdgeindexesareusedfromwithinAQLwhenperformingequalitylookupson_fromor_tovaluesinanedgecollections.Therearealsodedicatedfunctionstofindedgesgiventheir_fromor_tovaluesthatwillalwaysmakeuseoftheedgeindex:

db.collection.edges("<from-value>");

db.collection.edges("<to-value>");

db.collection.outEdges("<from-value>");

db.collection.outEdges("<to-value>");

db.collection.inEdges("<from-value>");

db.collection.inEdges("<to-value>");

Internally,theedgeindexisimplementedasahashindex,whichstorestheunionofall_fromand_toattributes.Itcanbeusedforequalitylookups,butnotforrangequeriesorforsorting.Edgeindexesareautomaticallycreatedforedgecollections.Itisnotpossibletocreateuser-definededgeindexes.However,itispossibletofreelyusethe_fromand_toattributesinuser-definedindexes.

Anedgeindexcannotbedroppedorchanged.

HashIndex

Ahashindexcanbeusedtoquicklyfinddocumentswithspecificattributevalues.Thehashindexisunsorted,soitsupportsequalitylookupsbutnorangequeriesorsorting.

IndexBasics

93

Ahashindexcanbecreatedononeormultipledocumentattributes.Ahashindexwillonlybeusedbyaqueryifallindexattributesarepresentinthesearchcondition,andifallattributesarecomparedusingtheequality(==)operator.HashindexesareusedfromwithinAQLandseveralqueryfunctions,e.g.byExample,firstExampleetc.

Hashindexescanoptionallybedeclaredunique,thendisallowingsavingthesamevalue(s)intheindexedattribute(s).Hashindexescanoptionallybesparse.

Thedifferenttypesofhashindexeshavethefollowingcharacteristics:

uniquehashindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Tryingtoinsertadocumentwiththesamekeyvalueasanalreadyexistingdocumentwillleadtoauniqueconstraintviolation.

Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Akeyvalueofnullmayonlyoccuronceintheindex,sothistypeofindexcannotbeusedforoptionalattributes.

Theuniqueoptioncanalsobeusedtoensurethatnoduplicateedgesarecreated,byaddingacombinedindexforthefields_fromand_totoanedgecollection.

unique,sparsehashindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Documentsinwhichatleastoneoftheindexattributesisnotsetorhasavalueofnullarenotincludedintheindex.Thistypeofindexcanbeusedtoensurethattherearenoduplicatekeysinthecollectionfordocumentswhichhavetheindexedattributesset.Astheindexwillexcludedocumentsforwhichtheindexedattributesarenullornotset,itcanbeusedforoptionalattributes.

non-uniquehashindex:alldocumentsinthecollectionwillbeindexed.Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Duplicatekeyvaluescanoccuranddonotleadtouniqueconstraintviolations.

non-unique,sparsehashindex:onlythosedocumentswillbeindexedthathavealltheindexedattributessettoavalueotherthannull.Itcanbeusedforoptionalattributes.

Theamortizedcomplexityoflookup,insert,update,andremovaloperationsinuniquehashindexesisO(1).

Non-uniquehashindexeshaveanamortizedcomplexityofO(1)forinsert,update,andremovaloperations.Thatmeansnon-uniquehashindexescanbeusedonattributeswithlowcardinality.

Ifahashindexiscreatedonanattributethatismissinginallormanyofthedocuments,thebehaviorisasfollows:

iftheindexissparse,thedocumentsmissingtheattributewillnotbeindexedandnotuseindexmemory.Thesedocumentswillnotinfluencetheupdateorremovalperformancefortheindex.

iftheindexisnon-sparse,thedocumentsmissingtheattributewillbecontainedintheindexwithakeyvalueofnull.

Hashindexessupportindexingarrayvaluesiftheindexattributenameisextendedwitha[*].

SkiplistIndex

Askiplistisasortedindexstructure.Itcanbeusedtoquicklyfinddocumentswithspecificattributevalues,forrangequeriesandforreturningdocumentsfromtheindexinsortedorder.SkiplistswillbeusedfromwithinAQLandseveralqueryfunctions,e.g.byExample,firstExampleetc.

Skiplistindexeswillbeusedforlookups,rangequeriesandsortingonlyifeitherallindexattributesareprovidedinaquery,orifaleftmostprefixoftheindexattributesisspecified.

Forexample,ifaskiplistindexiscreatedonattributesvalue1andvalue2,thefollowingfilterconditionscanusetheindex(note:the<=and>=operatorsareintentionallyomittedhereforthesakeofbrevity):

FILTERdoc.value1==...

FILTERdoc.value1<...

FILTERdoc.value1>...

FILTERdoc.value1>...&&doc.value1<...

FILTERdoc.value1==...&&doc.value2==...

FILTERdoc.value1==...&&doc.value2>...

FILTERdoc.value1==...&&doc.value2>...&&doc.value2<...

IndexBasics

94

Inordertouseaskiplistindexforsorting,theindexattributesmustbespecifiedintheSORTclauseofthequeryinthesameorderastheyappearintheindexdefinition.Skiplistindexesarealwayscreatedinascendingorder,buttheycanbeusedtoaccesstheindexedelementsinbothascendingordescendingorder.However,foracombinedindex(anindexonmultipleattributes)thisrequiresthatthesortordersinasinglequeryasspecifiedintheSORTclausemustbeeitherallascending(optionallyommittedasascendingisthedefault)oralldescending.

Forexample,iftheskiplistindexiscreatedonattributesvalue1andvalue2(inthisorder),thenthefollowingsortsclausescanusetheindexforsorting:

SORTvalue1ASC,value2ASC(anditsequivalentSORTvalue1,value2)SORTvalue1DESC,value2DESC

SORTvalue1ASC(anditsequivalentSORTvalue1)SORTvalue1DESC

Thefollowingsortclausescannotmakeuseoftheindexorder,andrequireanextrasortstep:

SORTvalue1ASC,value2DESC

SORTvalue1DESC,value2ASC

SORTvalue2(anditsequivalentSORTvalue2ASC)SORTvalue2DESC(becausefirstindexedattributevalue1isnotusedinsortclause)

Note:thelattertwosortclausescannotusetheindexbecausethesortclausedoesnotrefertoaleftmostprefixoftheindexattributes.

Skiplistscanoptionallybedeclaredunique,disallowingsavingthesamevalueintheindexedattribute.Theycanbesparseornon-sparse.

Thedifferenttypesofskiplistindexeshavethefollowingcharacteristics:

uniqueskiplistindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Tryingtoinsertadocumentwiththesamekeyvalueasanalreadyexistingdocumentwillleadtoauniqueconstraintviolation.

Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Akeyvalueofnullmayonlyoccuronceintheindex,sothistypeofindexcannotbeusedforoptionalattributes.

unique,sparseskiplistindex:alldocumentsinthecollectionmusthavedifferentvaluesfortheattributescoveredbytheuniqueindex.Documentsinwhichatleastoneoftheindexattributesisnotsetorhasavalueofnullarenotincludedintheindex.Thistypeofindexcanbeusedtoensurethattherearenoduplicatekeysinthecollectionfordocumentswhichhavetheindexedattributesset.Astheindexwillexcludedocumentsforwhichtheindexedattributesarenullornotset,itcanbeusedforoptionalattributes.

non-uniqueskiplistindex:alldocumentsinthecollectionwillbeindexed.Thistypeofindexisnotsparse.Documentsthatdonotcontaintheindexattributesorthathaveavalueofnullintheindexattribute(s)willstillbeindexed.Duplicatekeyvaluescanoccuranddonotleadtouniqueconstraintviolations.

non-unique,sparseskiplistindex:onlythosedocumentswillbeindexedthathavealltheindexedattributessettoavalueotherthannull.Itcanbeusedforoptionalattributes.

Theoperationalamortizedcomplexityforskiplistindexesislogarithmicallycorrelatedwiththenumberofdocumentsintheindex.

Skiplistindexessupportindexingarrayvaluesiftheindexattributenameisextendedwitha[*]`.

PersistentIndex

Thepersistentindexisasortedindexwithpersistence.Theindexentriesarewrittentodiskwhendocumentsarestoredorupdated.Thatmeanstheindexentriesdonotneedtoberebuiltfromthecollectiondatawhentheserverisrestartedortheindexedcollectionisinitiallyloaded.Thususingpersistentindexesmayreducecollectionloadingtimes.

Thepersistentindextypecanbeusedforsecondaryindexesatthemoment.Thatmeansthepersistentindexcurrentlycannotbemadetheonlyindexforacollection,becausetherewillalwaysbethein-memoryprimaryindexforthecollectioninaddition,andpotentiallymoreindexes(suchastheedgesindexforanedgecollection).

TheindeximplementationisusingtheRocksDBengine,anditprovideslogarithmiccomplexityforinsert,update,andremoveoperations.Asthepersistentindexisnotanin-memoryindex,itdoesnotstorepointersintotheprimaryindexasallthein-memoryindexesdo,butinsteaditstoresadocument'sprimarykey.Toretrieveadocumentviaapersistentindexviaanindexvaluelookup,therewillthereforebe

IndexBasics

95

anadditionalO(1)lookupintotheprimaryindextofetchtheactualdocument.

Asthepersistentindexissorted,itcanbeusedforpointlookups,rangequeriesandsortingoperations,butonlyifeitherallindexattributesareprovidedinaquery,orifaleftmostprefixoftheindexattributesisspecified.

GeoIndex

Userscancreateadditionalgeoindexesononeormultipleattributesincollections.Ageoindexisusedtofindplacesonthesurfaceoftheearthfast.

Thegeoindexstorestwo-dimensionalcoordinates.Itcanbecreatedoneithertwoseparatedocumentattributes(latitudeandlongitude)orasinglearrayattributethatcontainsbothlatitudeandlongitude.Latitudeandlongitudemustbenumericvalues.

Thegeoindexprovidesoperationstofinddocumentswithcoordinatesnearesttoagivencomparisoncoordinate,andtofinddocumentswithcoordinatesthatarewithinaspecifiableradiusaroundacomparisoncoordinate.

ThegeoindexisusedviadedicatedfunctionsinAQL,thesimplequeriesfunctionsanditisimplicitlyappliedwheninAQLaSORTorFILTERisusedwiththedistancefunction.Otherwiseitwillnotbeusedforothertypesofqueriesorconditions.

FulltextIndex

Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.Afulltextindexcanbecreatedonasingleattributeonly,andwillindexallwordscontainedindocumentsthathaveatextualvalueinthatattribute.Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.Wordsareindexedintheirlower-casedform.Theindexsupportscompletematchqueries(fullwords)andprefixqueries,plusbasiclogicaloperationssuchasand,orandnotforcombiningpartialresults.

Thefulltextindexissparse,meaningitwillonlyindexdocumentsforwhichtheindexattributeissetandcontainsastringvalue.Additionally,onlywordswithaconfigurableminimumlengthwillbeincludedintheindex.

ThefulltextindexisusedviadedicatedfunctionsinAQLorthesimplequeries,butwillnotbeenabledforothertypesofqueriesorconditions.

Indexingattributesandsub-attributes

Top-levelaswellasnestedattributescanbeindexed.Forattributesatthetoplevel,theattributenamesalonearerequired.Toindexasinglefield,passanarraywithasingleelement(stringoftheattributekey)tothefieldsparameteroftheensureIndex()method.Tocreateacombinedindexovermultiplefields,simplyaddmorememberstothefieldsarray:

//{name:"Smith",age:35}

db.posts.ensureIndex({type:"hash",fields:["name"]})

db.posts.ensureIndex({type:"hash",fields:["name","age"]})

Toindexsub-attributes,specifytheattributepathusingthedotnotation:

//{name:{last:"Smith",first:"John"}}

db.posts.ensureIndex({type:"hash",fields:["name.last"]})

db.posts.ensureIndex({type:"hash",fields:["name.last","name.first"]})

Indexingarrayvalues

Ifanindexattributecontainsanarray,ArangoDBwillstoretheentirearrayastheindexvaluebydefault.Accessingindividualmembersofthearrayviatheindexisnotpossiblethisway.

Tomakeanindexinserttheindividualarraymembersintotheindexinsteadoftheentirearrayvalue,aspecialarrayindexneedstobecreatedfortheattribute.Arrayindexescanbesetuplikeregularhashorskiplistindexesusingthecollection.ensureIndex()function.Tomakeahashorskiplistindexanarrayindex,theindexattributenameneedstobeextendedwith[*]whencreatingtheindexandwhenfilteringinanAQLqueryusingtheINoperator.

Thefollowingexamplecreatesanarrayhashindexonthetagsattributeinacollectionnamedposts:

IndexBasics

96

db.posts.ensureIndex({type:"hash",fields:["tags[*]"]});

db.posts.insert({tags:["foobar","baz","quux"]});

ThisarrayindexcanthenbeusedforlookingupindividualtagsvaluesfromAQLqueriesviatheINoperator:

FORdocINposts

FILTER'foobar'INdoc.tags

RETURNdoc

Itispossibletoaddthearrayexpansionoperator[*],butitisnotmandatory.Youmayuseittoindicatethatanarrayindexisused,itispurelycosmetichowever:

FORdocINposts

FILTER'foobar'INdoc.tags[*]

RETURNdoc

ThefollowingFILTERconditionswillnotusethearrayindex:

FILTERdoc.tagsANY=='foobar'

FILTERdoc.tagsANYIN'foobar'

FILTERdoc.tagsIN'foobar'

FILTERdoc.tags=='foobar'

FILTER'foobar'==doc.tags

Itisalsopossibletocreateanindexonsubattributesofarrayvalues.Thismakessenseiftheindexattributeisanarrayofobjects,e.g.

db.posts.ensureIndex({type:"hash",fields:["tags[*].name"]});

db.posts.insert({tags:[{name:"foobar"},{name:"baz"},{name:"quux"}]});

Thefollowingquerywillthenusethearrayindex(thisdoesrequirethearrayexpansionoperator):

FORdocINposts

FILTER'foobar'INdoc.tags[*].name

RETURNdoc

Ifyoustoreadocumenthavingthearraywhichdoescontainelementsnothavingthesubattributesthisdocumentwillalsobeindexedwiththevaluenull,whichinArangoDBisequaltoattributenotexisting.

ArangoDBsupportscreatingarrayindexeswithasingle[*]operatorperindexattribute.Forexample,creatinganindexasfollowsisnotsupported:

db.posts.ensureIndex({type:"hash",fields:["tags[*].name[*].value"]});

Arrayvalueswillautomaticallybede-duplicatedbeforebeinginsertedintoanarrayindex.Forexample,ifthefollowingdocumentisinsertedintothecollection,theduplicatearrayvaluebarwillbeinsertedonlyonce:

db.posts.insert({tags:["foobar","bar","bar"]});

Thisisdonetoavoidredudantstorageofthesameindexvalueforthesamedocument,whichwouldnotprovideanybenefit.

Ifanarrayindexisdeclaredunique,thede-duplicationofarrayvalueswillhappenbeforeinsertingthevaluesintotheindex,sotheaboveinsertoperationwithtwoidenticalvaluesbarwillnotnecessarilyfail

Itwillalwaysfailiftheindexalreadycontainsaninstanceofthebarvalue.However,ifthevaluebarisnotalreadypresentintheindex,thenthede-duplicationofthearrayvalueswilleffectivelyleadtobarbeinginsertedonlyonce.

Toturnoffthededuplicationofarrayvalues,itispossibletosetthededuplicateattributeonthearrayindextofalse.Thedefaultvaluefordeduplicateistruehowever,sode-duplicationwilltakeplaceifnotexplicitlyturnedoff.

db.posts.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:false});

IndexBasics

97

//willfailnow

db.posts.insert({tags:["foobar","bar","bar"]});

Ifanarrayindexisdeclaredandyoustoredocumentsthatdonothaveanarrayatthespecifiedattributethisdocumentwillnotbeinsertedintheindex.Hencethefollowingobjectswillnotbeindexed:


db.posts.insert({something:"else"});

db.posts.insert({tags:null});

db.posts.insert({tags:"thisisnoarray"});

db.posts.insert({tags:{content:[1,2,3]}});

Anarrayindexisabletoindexexplicitnullvalues.Whenqueriedfornullvalues,itwillonlyreturnthosedocumentshavingexplicitlynullstoredinthearray,itwillnotreturnanydocumentsthatdonothavethearrayatall.


db.posts.insert({tags:null})//Willnotbeindexed

db.posts.insert({tags:[]})//Willnotbeindexed

db.posts.insert({tags:[null]});//Willbeindexedfornull

db.posts.insert({tags:[null,1,2]});//Willbeindexedfornull,1and2

Declaringanarrayindexassparsedoesnothaveaneffectonthearraypartoftheindex,thisinparticularmeansthatexplicitnullvaluesarealsoindexedinthesparseversion.Ifanindexiscombinedfromanarrayandanormalattributethesparsitywillapplyfortheattributee.g.:

db.posts.ensureIndex({type:"hash",fields:["tags[*]","name"],sparse:true});

db.posts.insert({tags:null,name:"alice"})//Willnotbeindexed

db.posts.insert({tags:[],name:"alice"})//Willnotbeindexed

db.posts.insert({tags:[1,2,3]})//Willnotbeindexed

db.posts.insert({tags:[1,2,3],name:null})//Willnotbeindexed

db.posts.insert({tags:[1,2,3],name:"alice"})

//Willbeindexedfor[1,"alice"],[2,"alice"],[3,"alice"]

db.posts.insert({tags:[null],name:"bob"})

//Willbeindexedfor[null,"bob"]

PleasenotethatfilteringusingarrayindexesonlyworksfromwithinAQLqueriesandonlyifthequeryfiltersontheindexedattributeusingtheINoperator.Theothercomparisonoperators(==,!=,>,>=,<,<=,ANY,ALL,NONE)currentlycannotusearrayindexes.

Vertexcentricindexes

Asmentionedabove,themostimportantindexesforgraphsaretheedgeindexes,indexingthe_fromand_toattributesofedgecollections.Theyprovideveryquickaccesstoalledgesoriginatinginorarrivingatagivenvertex,whichallowstoquicklyfindallneighboursofavertexinagraph.

Inmanycasesonewouldliketorunmorespecificqueries,forexamplefindingamongsttheedgesoriginatinginagivenvertexonlythosewiththe20latesttimestamps.Exactlythisisachievedwith"vertexcentricindexes".Inasensethesearelocalizedindexesforanedgecollection,whichsitateverysinglevertex.

Technically,theyareimplementedinArangoDBasindexes,whichsortthecompleteedgecollectionfirstby_fromandthenbyotherattributes.Ifweforexamplehaveaskiplistindexontheattributes_fromandtimestampofanedgecollection,wecananswertheabovequestionveryquicklywithasinglerangelookupintheindex.

SinceArangoDB3.0onecancreatesortedindexes(type"skiplist"and"persistent")thatindexthespecialedgeattributes_fromor_toandadditionallyotherattributes.SinceArangoDB3.1,theseareusedingraphtraversals,whenappropriateFILTERstatementsarefoundbytheoptimizer.

Forexample,tocreateavertexcentricindexoftheabovetype,youwouldsimplydo

db.edges.ensureIndex({"type":"skiplist","fields":["_from","timestamp"]});

Then,querieslike

IndexBasics

98

FORv,e,pIN1..1OUTBOUND"V/1"edges

FILTERe.timestampALL>="2016-11-09"

RETURNp

willbeconsiderablyfasterincasetherearemanyedgesoriginatinginvertex"V/1"butonlyfewwitharecenttimestamp.

IndexBasics

99

WhichIndextousewhen

ArangoDBautomaticallyindexesthe_keyattributeineachcollection.Thereisnoneedtoindexthisattributeseparately.Pleasenotethatadocument's_idattributeisderivedfromthe_keyattribute,andisthusimplicitlyindexed,too.

ArangoDBwillalsoautomaticallycreateanindexon_fromand_toinanyedgecollection,meaningincomingandoutgoingconnectionscanbedeterminedefficiently.

Indextypes

Userscandefineadditionalindexesononeormultipledocumentattributes.SeveraldifferentindextypesareprovidedbyArangoDB.Theseindexeshavedifferentusagescenarios:

hashindex:providesquickaccesstoindividualdocumentsif(andonlyif)allindexedattributesareprovidedinthesearchquery.Theindexwillonlybeusedforequalitycomparisons.Itdoesnotsupportrangequeriesandcannotbeusedforsorting.

Thehashindexisagoodcandidateifallormostqueriesontheindexedattribute(s)areequalitycomparisons.TheuniquehashindexprovidesanamortizedcomplexityofO(1)forinsert,update,removeandlookupoperations.Thenon-uniquehashindexprovidesO(1)inserts,updatesandremoves,andwillallowlookingupdocumentsbyindexvaluewithamortizedO(n)complexity,withnbeingthenumberofdocumentswiththatindexvalue.

Anon-uniquehashindexonanoptionaldocumentattributeshouldbedeclaredsparsesothatitwillnotindexdocumentsforwhichtheindexattributeisnotset.

skiplistindex:skiplistskeeptheindexedvaluesinanorder,sotheycanbeusedforequalitylookups,rangequeriesandforsorting.Forhighselectivityattributes,skiplistindexeswillhaveahigheroverheadthanhashindexes.Forlowselectivityattributes,skiplistindexeswillbemoreefficientthannon-uniquehashindexes.

Additionally,skiplistindexesallowmoreusecases(e.g.rangequeries,sorting)thanhashindexes.Furthermore,theycanbeusedforlookupsbasedonaleftmostprefixoftheindexattributes.

persistentindex:apersistentindexbehavesmuchlikethesortedskiplistindex,exceptthatallindexvaluesarepersistedondiskanddonotneedtoberebuiltinmemorywhentheserverisrestartedortheindexedcollectionisreloaded.Theoperationsinapersistentindexhavelogarithmiccomplexity,butoperationshavemayhaveahigherconstantfactorthantheoperationsinaskiplistindex,becausethepersistentindexmayneedtomakeextraroundtripstotheprimaryindextofetchtheactualdocuments.

Apersistentindexcanbeusedforequalitylookups,rangequeriesandforsorting.Forhighselectivityattributes,persistentindexeswillhaveahigheroverheadthanskiplistorhashindexes.

Persistentindexesallowmoreusecases(e.g.rangequeries,sorting)thanhashindexes.Furthermore,theycanbeusedforlookupsbasedonaleftmostprefixoftheindexattributes.Incontrasttothein-memoryskiplistindexes,persistentindexesdonotneedtoberebuiltin-memorysotheydon'tinfluencetheloadingtimeofcollectionsasotherin-memoryindexesdo.

geoindex:thegeoindexprovidedbyArangoDBallowssearchingfordocumentswithinaradiusaroundatwo-dimensionalearthcoordinate(point),ortofinddocumentswithareclosesttoapoint.Documentcoordinatescaneitherbespecifiedintwodifferentdocumentattributesorinasingleattribute,e.g.

{"latitude":50.9406645,"longitude":6.9599115}

or

{"coords":[50.9406645,6.9599115]}

GeoindexeswillbeinvokedviaspecialfunctionsorAQLoptimization.TheoptimizationcanbetriggeredwhenacollectionwithgeoindexisenumeratedandaSORTorFILTERstatementisusedinconjunctionwiththedistancefunction.

fulltextindex:afulltextindexcanbeusedtoindexallwordscontainedinaspecificattributeofalldocumentsinacollection.Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.

Whichindextousewhen

100

Theindexsupportscompletematchqueries(fullwords)andprefixqueries.Fulltextindexeswillonlybeinvokedviaspecialfunctions.

Sparsevs.non-sparseindexes

Hashindexesandskiplistindexescanoptionallybecreatedsparse.Asparseindexdoesnotcontaindocumentsforwhichatleastoneoftheindexattributeisnotsetorcontainsavalueofnull.

Assuchdocumentsareexcludedfromsparseindexes,theymaycontainfewerdocumentsthantheirnon-sparsecounterparts.Thisenablesfasterindexingandcanleadtoreducedmemoryusageincasetheindexedattributedoesoccuronlyinsome,butnotalldocumentsofthecollection.Sparseindexeswillalsoreducethenumberofcollisionsinnon-uniquehashindexesincasenon-existingoroptionalattributesareindexed.

Inordertocreateasparseindex,anobjectwiththeattributesparsecanbeaddedtotheindexcreationcommands:

db.collection.ensureIndex({type:"hash",fields:["attributeName"],sparse:true});

db.collection.ensureIndex({type:"hash",fields:["attributeName1","attributeName2"],sparse:true});

db.collection.ensureIndex({type:"hash",fields:["attributeName"],unique:true,sparse:true});

db.collection.ensureIndex({type:"hash",fields:["attributeName1","attributeName2"],unique:true,sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName"],sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName1","attributeName2"],sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName"],unique:true,sparse:true});

db.collection.ensureIndex({type:"skiplist",fields:["attributeName1","attributeName2"],unique:true,sparse:true});

Whennotexplicitlyset,thesparseattributedefaultstofalsefornewindexes.Otherindexesthanhashandskiplistdonotsupportsparsity.

Assparseindexesmayexcludesomedocumentsfromthecollection,theycannotbeusedforalltypesofqueries.Sparsehashindexescannotbeusedtofinddocumentsforwhichatleastoneoftheindexedattributeshasavalueofnull.Forexample,thefollowingAQLquerycannotuseasparseindex,evenifonewascreatedonattributeattr:

FORdocIncollection

FILTERdoc.attr==null

RETURNdoc

Ifthelookupvalueisnon-constant,asparseindexmayormaynotbeused,dependingontheothertypesofconditionsinthequery.Iftheoptimizercansafelydeterminethatthelookupvaluecannotbenull,asparseindexmaybeused.Whenuncertain,theoptimizerwillnotmakeuseofasparseindexinaqueryinordertoproducecorrectresults.

Forexample,thefollowingqueriescannotuseasparseindexonattrbecausetheoptimizerwillnotknowbeforehandwhetherthevalueswhicharecomparedtodoc.attrwillincludenull:

FORdocIncollection

FILTERdoc.attr==SOME_FUNCTION(...)

RETURNdoc

FORotherINotherCollection

FORdocIncollection

FILTERdoc.attr==other.attr

RETURNdoc

Sparseskiplistindexescanbeusedforsortingiftheoptimizercansafelydetectthattheindexrangedoesnotincludenullforanyoftheindexattributes.

Notethatifyouintendtousejoinsitmaybeclevertousenon-sparsityandmaybeevenuniquenessforthatattribute,elseallitemscontainingthenullvaluewillmatchagainsteachotherandthusproducelargeresults.

Whichindextousewhen

101

IndexUtilizationInmostcasesArangoDBwilluseasingleindexpercollectioninagivenquery.AQLqueriescanusemorethanoneindexpercollectionwhenmultipleFILTERconditionsarecombinedwithalogicalORandthesecanbecoveredbyindexes.AQLquerieswilluseasingleindexpercollectionwhenFILTERconditionsarecombinedwithlogicalAND.

Creatingmultipleindexesondifferentattributesofthesamecollectionmaygivethequeryoptimizermorechoiceswhenpickinganindex.Creatingmultipleindexesondifferentattributescanalsohelpinspeedingupdifferentqueries,withFILTERconditionsondifferentattributes.

Itisoftenbeneficialtocreateanindexonmorethanjustoneattribute.Byaddingmoreattributestoanindex,anindexcanbecomemoreselectiveandthusreducethenumberofdocumentsthatqueriesneedtoprocess.

ArangoDB'sprimaryindexes,edgesindexesandhashindexeswillautomaticallyprovideselectivityestimates.Indexselectivityestimatesareprovidedinthewebinterface,thegetIndexes()returnvalueandintheexplain()outputforagivenquery.

Themoreselectiveanindexis,themoredocumentsitwillfilteronaverage.Theindexselectivityestimatesarethereforeusedbytheoptimizerwhencreatingqueryexecutionplanswhentherearemultipleindexestheoptimizercanchoosefrom.Theoptimizerwillthenselectacombinationofindexeswiththelowestestimatedtotalcost.Ingeneral,theoptimizerwillpicktheindexeswiththehighestestimatedselectivity.

Sparseindexesmayormaynotbepickedbytheoptimizerinaquery.Assparseindexesdonotcontainnullvalues,theywillnotbeusedforqueriesiftheoptimizercannotsafelydeterminewhetheraFILTERconditionincludesnullvaluesfortheindexattributes.Theoptimizerpolicyistoproducecorrectresults,regardlessofwhetherorwhichindexisusedtosatisfyFILTERconditions.Ifitisunsureaboutwhetherusinganindexwillviolatethepolicy,itwillnotmakeuseoftheindex.

Troubleshooting

WhenindoubtaboutwhetherandwhichindexeswillbeusedforexecutingagivenAQLquery,clicktheExplainbuttoninthewebinterfaceintheQueriesvieworusetheexplain()methodforthestatementasfollows(fromtheArangoShell):

varquery="FORdocINcollectionFILTERdoc.value>42RETURNdoc";

varstmt=db._createStatement(query);

stmt.explain();

Theexplain()commandwillreturnadetailedJSONrepresentationofthequery'sexecutionplan.TheJSONexplainoutputisintendedtobeusedbycode.Togetahuman-readableandmuchmorecompactexplanationofthequery,thereisanexplainertool:

varquery="FORdocINcollectionFILTERdoc.value>42RETURNdoc";

require("@arangodb/aql/explainer").explain(query);

Ifanyoftheexplainmethodsshowsthataqueryisnotusingindexes,thefollowingstepsmayhelp:

checkiftheattributenamesinthequeryarecorrectlyspelled.Inaschema-freedatabase,documentsinthesamecollectioncanhavevaryingstructures.Thereisnosuchthingasanon-existingattributeerror.Aquerythatreferstoattributenamesnotpresentinanyofthedocumentswillnotreturnanerror,andobviouslywillnotbenefitfromindexes.

checkthereturnvalueofthegetIndexes()methodforthecollectionsusedinthequeryandvalidatethatindexesareactuallypresentontheattributesusedinthequery'sfilterconditions.

ifindexesarepresentbutnotusedbythequery,theindexesmayhavethewrongtype.Forexample,ahashindexwillonlybeusedforequalitycomparisons(i.e.==)butnotforothercomparisontypessuchas<,<=,>,>=.Additionallyhashindexeswillonlybeusedifalloftheindexattributesareusedinthequery'sFILTERconditions.AskiplistindexwillonlybeusedifatleastitsfirstattributeisusedinaFILTERcondition.Ifadditionallyoftheskiplistindexattributesarespecifiedinthequery(fromleft-to-right),theymayalsobeusedandallowtofiltermoredocuments.

usingindexedattributesasfunctionparametersorinarbitraryexpressionswilllikelyleadtotheindexontheattributenotbeingused.Forexample,thefollowingquerieswillnotuseanindexonvalue:

IndexUtilization

102

FORdocINcollectionFILTERTO_NUMBER(doc.value)==42RETURNdoc

FORdocINcollectionFILTERdoc.value-1==42RETURNdoc

Inthesecasesthequeriesshouldberewrittensothatonlytheindexattributeispresentononesideoftheoperator,oradditionalfiltersandindexesshouldbeusedtorestricttheamountofdocumentsotherwise.

certainAQLfunctionssuchasWITHIN()orFULLTEXT()doutilizeindexesinternally,buttheiruseisnotmentionedinthequeryexplanationforfunctionsingeneral.Thesefunctionswillraisequeryerrors(atruntime)ifnosuitableindexispresentforthecollectioninquestion.

thequeryoptimizerwillingeneralpickoneindexpercollectioninaquery.ItcanpickmorethanoneindexpercollectioniftheFILTERconditioncontainsmultiplebranchescombinedwithlogicalOR.Forexample,thefollowingqueriescanuseindexes:

FORdocINcollectionFILTERdoc.value1==42||doc.value1==23RETURNdoc


FORdocINcollectionFILTERdoc.value1<42||doc.value2>23RETURNdoc

ThetwoORsinthefirstquerywillbeconvertedtoanINlist,andifthereisasuitableindexonvalue1,itwillbeused.Thesecondqueryrequirestwoseparateindexesonvalue1andvalue2andwillusethemifpresent.Thethirdquerycanusetheindexesonvalue1andvalue2whentheyaresorted.

IndexUtilization

103

WorkingwithIndexes

IndexIdentifiersandHandles

Anindexhandleuniquelyidentifiesanindexinthedatabase.Itisastringandconsistsofthecollectionnameandanindexidentifierseparatedbya/.Theindexidentifierpartisanumericvaluethatisauto-generatedbyArangoDB.

Aspecificindexofacollectioncanbeaccessedusingitsindexhandleorindexidentifierasfollows:

db.collection.index("<index-handle>");

db.collection.index("<index-identifier>");

db._index("<index-handle>");

Forexample:Assumethattheindexhandle,whichisstoredinthe_idattributeoftheindex,isdemo/362549736andtheindexwascreatedinacollectionnameddemo.Thenthisindexcanbeaccessedas:

db.demo.index("demo/362549736");

Becausetheindexhandleisuniquewithinthedatabase,youcanleaveoutthecollectionandusetheshortcut:

db._index("demo/362549736");

CollectionMethods

Listingallindexesofacollection

returnsinformationabouttheindexesgetIndexes()

Returnsanarrayofallindexesdefinedforthecollection.

Notethat_keyimplicitlyhasanindexassignedtoit.

arangosh>db.test.ensureHashIndex("hashListAttribute",

........>"hashListSecondAttribute.subAttribute");

arangosh>db.test.getIndexes();


Creatinganindex

IndexesshouldbecreatedusingthegeneralmethodensureIndex.Thismethodobsoletesthespecializedindex-specificmethodsensureHashIndex,ensureSkiplist,ensureUniqueConstraintetc.

ensuresthatanindexexistscollection.ensureIndex(index-description)

Ensuresthatanindexaccordingtotheindex-descriptionexists.Anewindexwillbecreatedifnoneexistswiththegivendescription.

Theindex-descriptionmustcontainatleastatypeattribute.Otherattributesmaybenecessary,dependingontheindextype.

typecanbeoneofthefollowingvalues:

hash:hashindexskiplist:skiplistindexfulltext:fulltextindexgeo1:geoindex,withoneattributegeo2:geoindex,withtwoattributes

WorkingwithIndexes

104

sparsecanbetrueorfalse.

Forhash,andskiplistthesparsitycanbecontrolled,fulltextandgeoaresparsebydefinition.

uniquecanbetrueorfalseandissupportedbyhashorskiplist

Callingthismethodreturnsanindexobject.WhetherornottheindexobjectexistedbeforethecallisindicatedinthereturnattributeisNewlyCreated.

deduplicatecanbetrueorfalseandissupportedbyarrayindexesoftypehashorskiplist.Itcontrolswhetherinsertingduplicateindexvaluesfromthesamedocumentintoauniquearrayindexwillleadtoauniqueconstrainterrorornot.Thedefaultvalueistrue,soonlyasingleinstanceofeachnon-uniqueindexvaluewillbeinsertedintotheindexperdocument.Tryingtoinsertavalueintotheindexthatalreadyexistsintheindexwillalwaysfail,regardlessofthevalueofthisattribute.

Examples

arangosh>db.test.ensureIndex({type:"hash",fields:["a"],sparse:true});

arangosh>db.test.ensureIndex({type:"hash",fields:["a","b"],unique:true});


Droppinganindex

dropsanindexcollection.dropIndex(index)

Dropstheindex.Iftheindexdoesnotexist,thenfalseisreturned.Iftheindexexistedandwasdropped,thentrueisreturned.Notethatyoucannotdropsomespecialindexes(e.g.theprimaryindexofacollectionortheedgeindexofanedgecollection).

collection.dropIndex(index-handle)

Sameasabove.Insteadofanindexanindexhandlecanbegiven.

arangosh>db.example.ensureSkiplist("a","b");

arangosh>varindexInfo=db.example.getIndexes();

arangosh>indexInfo;

arangosh>db.example.dropIndex(indexInfo[0])

arangosh>db.example.dropIndex(indexInfo[1].id)

arangosh>indexInfo=db.example.getIndexes();


LoadIndexesintoMemory

LoadsallindexesofthiscollectionintoMemory.collection.loadIndexesIntoMemory()

Thisfunctiontriestocacheallindexentriesofthiscollectionintothemainmemory.Thereforeititeratesoverallindexesofthecollectionandstorestheindexedvalues,nottheentiredocumentdata,inmemory.Alllookupsthatcouldbefoundinthecachearemuchfasterthanlookupsnotstoredinthecachesoyougetaniceperformanceboost.Itisalsoguaranteedthatthecacheisconsistentwiththestoreddata.

ForthetimebeingthisfunctionisonlyusefulonRocksDBstorageengine,asinMMFilesengineallindexesareinmemoryanyways.

OnRocksDBthisfunctionhonorsallmemorylimits,iftheindexesyouwanttoloadaresmallerthanyourmemorylimitthisfunctionguaranteesthatmostindexvaluesarecached.Iftheindexislargerthanyourmemorylimitthisfunctionwillfillupvaluesuptothislimitandforthetimebeingthereisnowaytocontrolwhichindexesofthecollectionshouldhavepriorityoverothers.

arangosh>db.example.loadIndexesIntoMemory();

{

"result":true

}

WorkingwithIndexes

105

DatabaseMethods

Fetchinganindexbyhandle

findsanindexdb._index(index-handle)

Returnstheindexwithindex-handleornullifnosuchindexexists.

arangosh>db.example.ensureIndex({type:"skiplist",fields:["a","b"]});

arangosh>varindexInfo=db.example.getIndexes().map(function(x){returnx.id;});

arangosh>indexInfo;

arangosh>db._index(indexInfo[0])

arangosh>db._index(indexInfo[1])


Droppinganindex

dropsanindexdb._dropIndex(index)

Dropstheindex.Iftheindexdoesnotexist,thenfalseisreturned.Iftheindexexistedandwasdropped,thentrueisreturned.

db._dropIndex(index-handle)

Dropstheindexwithindex-handle.


arangosh>varindexInfo=db.example.getIndexes();

arangosh>indexInfo;

arangosh>db._dropIndex(indexInfo[0])

arangosh>db._dropIndex(indexInfo[1].id)

arangosh>indexInfo=db.example.getIndexes();


Revalidatingwhetheranindexisused

findsanindex

Soyou'vecreatedanindex,andsinceitsmaintainanceisn'tforfree,youdefinitelywanttoknowwhetheryourquerycanutilizeit.

Youcanuseexplaintoverifywhetherskiplistsorhashindexesareused(ifyouomitcolors:falseyouwillgetnicecolorsinArangoShell):

arangosh>varexplain=require("@arangodb/aql/explainer").explain;


arangosh>explain("FORdocINexampleFILTERdoc.a<23RETURNdoc",{colors:false});


WorkingwithIndexes

106

HashIndexes

IntroductiontoHashIndexes

Itispossibletodefineahashindexononeormoreattributes(orpaths)ofadocument.ThishashindexisthenusedinqueriestolocatedocumentsinO(1)operations.Ifthehashindexisunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.

Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.

AccessingHashIndexesfromtheShell

UniqueHashIndexes

Ensuresthatauniqueconstraintexists:collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],unique:true})

Createsauniquehashindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillwillfailiftheattributeuniquenessisviolated.

Tocreateasparseuniqueindex,setthesparseattributetotrue:

collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],unique:true,sparse:true})

Incasethattheindexwassuccessfullycreated,theindexidentifierisreturned.

Non-existingattributeswilldefaulttonull.Inasparseindexalldocumentswillbeexcludedfromtheindexforwhichallspecifiedindexattributesarenull.Suchdocumentswillnotbetakenintoaccountforuniquenesschecks.

Inanon-sparseindex,alldocumentsregardlessofnull-attributeswillbeindexedandwillbetakenintoaccountforuniquenesschecks.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetails,includingtheindex-identifier,isreturned.

arangosh>db.test.ensureIndex({type:"hash",fields:["a","b.c"],unique:true});

arangosh>db.test.save({a:1,b:{c:1}});

arangosh>db.test.save({a:1,b:{c:1}});

arangosh>db.test.save({a:1,b:{c:null}});

arangosh>db.test.save({a:1});


Non-uniqueHashIndexes

Ensuresthatanon-uniquehashindexexists:collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"]})

Createsanon-uniquehashindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.


collection.ensureIndex({type:"hash",fields:["field1",...,"fieldn"],sparse:true})


arangosh>db.test.ensureIndex({type:"hash",fields:["a"]});


HashIndexes

107


arangosh>db.test.save({a:null});


HashArrayIndexes

Ensuresthatahasharrayindexexists(non-unique):collection.ensureIndex({type:"hash",fields:["field1[*]",...,"fieldn[*]"]})

Createsanon-uniquehasharrayindexfortheindividualelementsofthearrayattributesfield1[*],...fieldn[*]foundinthedocuments.Atleastoneattributepathhastobegiven.Theindexalwaystreatstheindexedarraysassparse.

Itispossibletocombinearrayindexingwithstandardindexing:collection.ensureIndex({type:"hash",fields:["field1[*]","field2"]})


arangosh>db.test.ensureIndex({type:"hash",fields:["a[*]"]});

arangosh>db.test.save({a:[1,2]});




Ensureuniquenessofrelationsinedgecollections

Itispossibletocreatesecondaryindexesusingtheedgeattributes_fromand_to,startingwithArangoDB3.0.Acombinedindexoverbothfieldstogetherwiththeuniqueoptionenabledcanbeusedtopreventduplicaterelationsfrombeingcreated.

Forexample,adocumentcollectionvertsmightcontainverticeswiththedocumenthandlesverts/A,verts/Bandverts/C.Relationsbetweenthesedocumentscanbestoredinanedgecollectionedgesforinstance.Now,youmaywanttomakesurethatthevertexverts/Aisneverlinkedtoverts/Bbyanedgemorethanonce.Thiscanbeachievedbyaddingaunique,non-sparsehashindexforthefields_fromand_to:

db.edges.ensureIndex({type:"hash",fields:["_from","_to"],unique:true});

Creatinganedge{_from:"verts/A",_to:"verts/B"}inedgeswillbeaccepted,butonlyonce.AnotherattempttostoreanedgewiththerelationA→Bwillberejectedbytheserverwithauniqueconstraintviolatederror.Thisincludesupdatestothe_fromand_tofields.

NotethataddingarelationB→Aisstillpossible,soisA→AandB→B,becausetheyarealldifferentrelationsinadirectedgraph.Eachonecanonlyoccuroncehowever.

HashIndexes

108

Skiplists

IntroductiontoSkiplistIndexes

ThisisanintroductiontoArangoDB'sskiplists.

Itispossibletodefineaskiplistindexononeormoreattributes(orpaths)ofdocuments.Thisskiplististhenusedinqueriestolocatedocumentswithinagivenrange.Iftheskiplistisdeclaredunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.

Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheskiplistindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.

AccessingSkiplistIndexesfromtheShell

UniqueSkiplistIndex

Ensuresthatauniqueskiplistindexexists:collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],unique:true})

Createsauniqueskiplistindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillfailiftheattributeuniquenessisviolated.


collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],unique:true,sparse:true})

Inasparseindexalldocumentswillbeexcludedfromtheindexthatdonotcontainatleastoneofthespecifiedindexattributesorthathaveavalueofnullinanyofthespecifiedindexattributes.Suchdocumentswillnotbeindexed,andnotbetakenintoaccountforuniquenesschecks.

Inanon-sparseindex,thesedocumentswillbeindexed(fornon-presentindexedattributes,avalueofnullwillbeused)andwillbetakenintoaccountforuniquenesschecks.


arangosh>db.ids.ensureIndex({type:"skiplist",fields:["myId"],unique:true});

arangosh>db.ids.save({"myId":123});





arangosh>db.ids.ensureIndex({type:"skiplist",fields:["name.first","name.last"],

unique:true});

arangosh>db.ids.save({"name":{"first":"hans","last":"hansen"}});

arangosh>db.ids.save({"name":{"first":"jens","last":"jensen"}});

arangosh>db.ids.save({"name":{"first":"hans","last":"jensen"}});



Non-uniqueSkiplistIndex

Skiplists

109

Ensuresthatanon-uniqueskiplistindexexists:collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"]})

Createsanon-uniqueskiplistindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Tocreateasparsenon-uniqueindex,setthesparseattributetotrue.

collection.ensureIndex({type:"skiplist",fields:["field1",...,"fieldn"],sparse:true})


arangosh>db.names.ensureIndex({type:"skiplist",fields:["first"]});

arangosh>db.names.save({"first":"Tim"});

arangosh>db.names.save({"first":"Tom"});

arangosh>db.names.save({"first":"John"});




SkiplistArrayIndex

Ensuresthataskiplistarrayindexexists(non-unique):collection.ensureIndex({type:"skiplist",fields:["field1[*]",...,"fieldn[*]"]})

Createsanon-uniqueskiplistarrayindexfortheindividualelementsofthearrayattributesfield1[*],...fieldn[*]foundinthedocuments.Atleastoneattributepathhastobegiven.Theindexalwaystreatstheindexedarraysassparse.

Itispossibletocombinearrayindexingwithstandardindexing:collection.ensureIndex({type:"skiplist",fields:["field1[*]","field2"]})


arangosh>db.test.ensureIndex({type:"skiplist",fields:["a[*]"]});





Querybyexampleusingaskiplistindex

Constructsaquery-by-exampleusingaskiplistindex:collection.byExample(example)

Selectsalldocumentsfromthecollectionthatmatchthespecifiedexampleandreturnsacursor.Askiplistindexwillbeusedifpresent.



{"a":{"c":1}}


{"a":{"c":1},"b":1}


{"a":{"c":1,"b":1}}

Skiplists

110

willnot.

However,ifyouuse

{"a.c":1},


{"a":{"c":1},"b":1}

and

{"a":{"c":1,"b":1}}

willmatch.

Skiplists

111

Persistentindexes

IntroductiontoPersistentIndexes

ThisisanintroductiontoArangoDB'spersistentindexes.

Itispossibletodefineapersistentindexononeormoreattributes(orpaths)ofdocuments.Theindexisthenusedinqueriestolocatedocumentswithinagivenrange.Iftheindexisdeclaredunique,thennotwodocumentsareallowedtohavethesamesetofattributevalues.

Creatinganewdocumentorupdatingadocumentwillfailiftheuniquenessisviolated.Iftheindexisdeclaredsparse,adocumentwillbeexcludedfromtheindexandnouniquenesscheckswillbeperformedifanyindexattributevalueisnotsetorhasavalueofnull.

AccessingPersistentIndexesfromtheShell

ensuresthatauniquepersistentindexexistscollection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"],unique:true})

Createsauniquepersistentindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Alldocumentsinthecollectionmustdifferintermsoftheindexedattributes.Creatinganewdocumentorupdatinganexistingdocumentwillwillfailiftheattributeuniquenessisviolated.


collection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"],unique:true,sparse:true})

Inasparseindexalldocumentswillbeexcludedfromtheindexthatdonotcontainatleastoneofthespecifiedindexattributesorthathaveavalueofnullinanyofthespecifiedindexattributes.Suchdocumentswillnotbeindexed,andnotbetakenintoaccountforuniquenesschecks.

Inanon-sparseindex,thesedocumentswillbeindexed(fornon-presentindexedattributes,avalueofnullwillbeused)andwillbetakenintoaccountforuniquenesschecks.


arangosh>db.ids.ensureIndex({type:"persistent",fields:["myId"],unique:true});






arangosh>db.ids.ensureIndex({type:"persistent",fields:["name.first","name.last"],

unique:true});


arangosh>db.ids.save({"name":{"first":"jens","last":"jensen"}});

arangosh>db.ids.save({"name":{"first":"hans","last":"jensen"}});


showexecutionresultsensuresthatanon-uniquepersistentindexexistscollection.ensureIndex({type:"persistent",fields:["field1",...,"fieldn"]})

Createsanon-uniquepersistentindexonalldocumentsusingfield1,...fieldnasattributepaths.Atleastoneattributepathhastobegiven.Theindexwillbenon-sparsebydefault.

Persistent

112

Tocreateasparseuniqueindex,setthesparseattributetotrue.


arangosh>db.names.ensureIndex({type:"persistent",fields:["first"]});



arangosh>db.names.save({"first":"John"});




Querybyexampleusingapersistentindex

constructsaquery-by-exampleusingapersistentindexcollection.byExample(example)

Selectsalldocumentsfromthecollectionthatmatchthespecifiedexampleandreturnsacursor.Apersistentindexwillbeusedifpresent.



{"a":{"c":1}}


{"a":{"c":1},"b":1}


{"a":{"c":1,"b":1}}

willnot.

However,ifyouuse

{"a.c":1},


{"a":{"c":1},"b":1}

and

{"a":{"c":1,"b":1}}

willmatch.

PersistentIndexesandServerLanguageTheorderofindexentriesinpersistentindexesadherestotheconfiguredserverlanguage.If,however,theserverisrestartedwithadifferentlanguagesettingaswhenthepersistentindexwascreated,notalldocumentsmaybereturnedanymoreandthesortorderofthosewhicharereturnedcanbewrong(wheneverthepersistentindexisconsulted).

Tofixpersistentindexesafteralanguagechange,deleteandre-createthem.Skiplistindexesarenotaffected,becausetheyarenotpersistedandautomaticallyrebuiltoneveryserverstart.

Persistent

113

Persistent

114

FulltextindexesThisisanintroductiontoArangoDB'sfulltextindexes.

IntroductiontoFulltextIndexes

Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.

Afulltextindexcanbedefinedononeattributeonly,andwillincludeallwordscontainedindocumentsthathaveatextualvalueintheindexattribute.SinceArangoDB2.6theindexwillalsoincludewordsfromtheindexattributeiftheindexattributeisanarrayofstrings,oranobjectwithstringvaluemembers.

Forexample,givenafulltextindexonthetranslationsattributeandthefollowingdocuments,thensearchingforлисаusingthefulltextindexwouldreturnonlythefirstdocument.SearchingfortheindexfortheexactstringFoxwouldreturnthefirsttwodocuments,andsearchingforprefix:Foxwouldreturnallthreedocuments:

{translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}}

{translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"}

{translations:["ArangoDB","document","database","Foxx"]}

Notethatdeepernestedobjectsareignored.Forexample,afulltextindexontranslationswouldindexFuchs,butnotfox,giventhefollowingdocumentstructure:

{translations:{en:{US:"fox"},de:"Fuchs"}

Ifyouneedtosearchacrossmultiplefieldsand/ornestedobjects,youmaywriteallthestringsintoaspecialattribute,whichyouthencreatetheindexon(itmightbenecessarytocleanthestringsfirst,e.g.removelinebreaksandstripcertainwords).

Iftheindexattributeisneitherastring,anobjectoranarray,itscontentswillnotbeindexed.Whenindexingthecontentsofanarrayattribute,anarraymemberwillonlybeincludedintheindexifitisastring.Whenindexingthecontentsofanobjectattribute,anobjectmembervaluewillonlybeincludedintheindexifitisastring.Otherdatatypesareignoredandnotindexed.

Currently,fulltextindexesarenotyetsupportedwiththeRocksDBstorageengine.ThusthefunctionFULLTEXT()willbeunavailablewhenusingthisstorageengine.Tousefulltextindexes,pleaseusetheMMFilesstorageengineforthetimebeing.

AccessingFulltextIndexesfromtheShell

Ensuresthatafulltextindexexists:

collection.ensureIndex({type:"fulltext",fields:["field"],minLength:minLength})

Createsafulltextindexonalldocumentsonattributefield.

Fulltextindexesareimplicitlysparse:alldocumentswhichdonothavethespecifiedfieldattributeorthathaveanon-qualifyingvalueintheirfieldattributewillbeignoredforindexing.

Onlyasingleattributecanbeindexed.Specifyingmultipleattributesisunsupported.

TheminimumlengthofwordsthatareindexedcanbespecifiedviatheminLengthparameter.WordsshorterthanminLengthcharacterswillnotbeindexed.minLengthhasadefaultvalueof2,butthisvaluemightbechangedinfutureversionsofArangoDB.Itisthusrecommendedtoexplicitlyspecifythisvalue.

Incasethattheindexwassuccessfullycreated,anobjectwiththeindexdetailsisreturned.

arangosh>db.example.ensureIndex({type:"fulltext",fields:["text"],minLength:3});

arangosh>db.example.save({text:"thequickbrown",b:{c:1}});

arangosh>db.example.save({text:"quickbrownfox",b:{c:2}});

arangosh>db.example.save({text:"brownfoxjums",b:{c:3}});

arangosh>db.example.save({text:"foxjumpsover",b:{c:4}});

FulltextIndexes

115

arangosh>db.example.save({text:"jumpsoverthe",b:{c:5}});

arangosh>db.example.save({text:"overthelazy",b:{c:6}});

arangosh>db.example.save({text:"thelazydog",b:{c:7}});

arangosh>db._query("FORdocumentINFULLTEXT(example,'text','the')RETURNdocument");

showexecutionresultsLooksupafulltextindex:

collection.lookupFulltextIndex(attribute,minLength)

Checkswhetherafulltextindexonthegivenattributeattributeexists.

FulltextAQLFunctions

FulltextAQLfunctionsaredetailedinFulltextfunctions.

FulltextIndexes

116

GeoIndexes

IntroductiontoGeoIndexes

ThisisanintroductiontoArangoDB'sgeoindexes.

AQL'sgeographicfeaturesaredescribedinGeofunctions.

ArangoDBusesHilbertcurvestoimplementgeo-spatialindexes.Seethisblogfordetails.

Ageo-spatialindexassumesthatthelatitudeisbetween-90and90degreeandthelongitudeisbetween-180and180degree.Ageoindexwillignorealldocumentswhichdonotfulfilltheserequirements.

AccessingGeoIndexesfromtheShell

ensuresthatageoindexexistscollection.ensureIndex({type:"geo",fields:["location"]})

Createsageo-spatialindexonalldocumentsusinglocationaspathtothecoordinates.Thevalueoftheattributehastobeanarraywithatleasttwonumericvalues.Thearraymustcontainthelatitude(firstvalue)andthelongitude(secondvalue).

Alldocuments,whichdonothavetheattributepathorhaveanon-conformingvalueinitareexcludedfromtheindex.

Ageoindexisimplicitlysparse,andthereisnowaytocontrolitssparsity.


Tocreateageoindexonanarrayattributethatcontainslongitudefirst,setthegeoJsonattributetotrue.ThiscorrespondstotheformatdescribedinRFC7946Position

collection.ensureIndex({type:"geo",fields:["location"],geoJson:true})

Tocreateageo-spatialindexonalldocumentsusinglatitudeandlongitudeasseparateattributepaths,twopathsneedtobespecifiedinthefieldsarray:

collection.ensureIndex({type:"geo",fields:["latitude","longitude"]})


Examples

Createageoindexforanarrayattribute:

arangosh>db.geo.ensureIndex({type:"geo",fields:["loc"]});


........>for(j=-180;j<=180;j+=10){

........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});

........>}

........>}

arangosh>db.geo.count();

arangosh>db.geo.near(0,0).limit(3).toArray();

arangosh>db.geo.near(0,0).count();

showexecutionresultsCreateageoindexforahasharrayattribute:

arangosh>db.geo2.ensureIndex({type:"geo",fields:["location.latitude",

"location.longitude"]});


........>for(j=-180;j<=180;j+=10){

........>db.geo2.save({name:"Name/"+i+"/"+j,location:{latitude:i,

longitude:j}});

GeoIndexes

117

https://www.arangodb.com/2012/03/31/using-hilbert-curves-and-polyhedrons-for-geo-indexing

https://tools.ietf.org/html/rfc7946#section-3.1.1

........>}

........>}

arangosh>db.geo2.near(0,0).limit(3).toArray();

showexecutionresultsUseGeoIndexwithAQLSORTstatement:

arangosh>db.geoSort.ensureIndex({type:"geo",fields:["latitude","longitude"]});


........>for(j=-180;j<=180;j+=10){

........>db.geoSort.save({name:"Name/"+i+"/"+j,latitude:i,longitude

:j});

........>}

........>}

arangosh>varquery="FORdocingeoSortSORTDISTANCE(doc.latitude,doc.longitude,0,0)

LIMIT5RETURNdoc"

arangosh>db._explain(query,{},{colors:false});

arangosh>db._query(query);

showexecutionresultsUseGeoIndexwithAQLFILTERstatement:

arangosh>db.geoFilter.ensureIndex({type:"geo",fields:["latitude","longitude"]});


........>for(j=-180;j<=180;j+=10){

........>db.geoFilter.save({name:"Name/"+i+"/"+j,latitude:i,

longitude:j});

........>}

........>}

arangosh>varquery="FORdocingeoFilterFILTERDISTANCE(doc.latitude,doc.longitude,

0,0)<2000RETURNdoc"

arangosh>db._explain(query,{},{colors:false});

arangosh>db._query(query);

showexecutionresultsconstructsageoindexselectioncollection.geo(location-attribute)

Looksupageoindexdefinedonattributelocation_attribute.

Returnsageoindexobjectifanindexwasfound.Thenearorwithinoperatorscanthenbeusedtoexecuteageo-spatialqueryonthisparticularindex.

Thisisusefulforcollectionswithmultipledefinedgeoindexes.

collection.geo(location_attribute,true)

Looksupageoindexonacompoundattributelocation_attribute.


collection.geo(latitude_attribute,longitude_attribute)

Looksupageoindexdefinedonthetwoattributeslatitude_attributeandlongitude-attribute.


Note:thismethodisnotyetsupportedbytheRocksDBstorageengine.

GeoIndexes

118

Note:thegeosimplequeryhelperfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforrunninggeoqueriesistousetheirAQLequivalents.

Examples

Assumeyouhavealocationstoredaslistintheattributehomeandadestinationstoredintheattributework.Thenyoucanusethegeooperatortoselectwhichgeo-spatialattributes(andthuswhichindex)touseinanearquery.


........>for(j=-180;j<=180;j+=10){

........>db.complex.save({name:"Name/"+i+"/"+j,

........>home:[i,j],

........>work:[-i,-j]});

........>}

........>}

........>

arangosh>db.complex.near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["home"]});

arangosh>db.complex.near(0,170).limit(5).toArray();

arangosh>db.complex.geo("work").near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["work"]});

arangosh>db.complex.geo("work").near(0,170).limit(5).toArray();

showexecutionresultsconstructsanearqueryforacollectioncollection.near(latitude,longitude)

Thereturnedlistissortedaccordingtothedistance,withthenearestdocumenttothecoordinate(latitude,longitude)comingfirst.Ifthereareneardocumentsofequaldistance,documentsarechosenrandomlyfromthissetuntilthelimitisreached.Itispossibletochangethelimitusingthelimitoperator.

Inordertousethenearoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.

Note:neardoesnotsupportnegativeskips.//However,youcanstilluselimitfollowedtoskip.

collection.near(latitude,longitude).limit(limit)

Limitstheresulttolimitdocumentsinsteadofthedefault100.

Note:Unlikewithmultipleexplicitlimits,limitwillraisetheimplicitdefaultlimitimposedbywithin.

collection.near(latitude,longitude).distance()

Thiswilladdanattributedistancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.

collection.near(latitude,longitude).distance(name)

Thiswilladdanattributenametoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.


Note:thenearsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthenearoperatoristousetheAQLNEARfunctioninanAQLqueryasfollows:

FORdocINNEAR(@@collection,@latitude,@longitude,@limit)

RETURNdoc

Examples

Togetthenearesttwolocations:

GeoIndexes

119


arangosh>for(vari=-90;i<=90;i+=10){

........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({

........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}


showexecutionresultsIfyouneedthedistanceaswell,thenyoucanusethedistanceoperator:



........>for(varj=-180;j<=180;j+=10){


........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}

arangosh>db.geo.near(0,0).distance().limit(2).toArray();

showexecutionresultsconstructsawithinqueryforacollectioncollection.within(latitude,longitude,radius)

Thiswillfindalldocumentswithinagivenradiusaroundthecoordinate(latitude,longitude).Thereturnedarrayissortedbydistance,beginningwiththenearestdocument.

Inordertousethewithinoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.

collection.within(latitude,longitude,radius).distance()

Thiswilladdanattribute_distancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.

collection.within(latitude,longitude,radius).distance(name)



Note:thewithinsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthewithinoperatoristousetheAQLWITHINfunctioninanAQLqueryasfollows:

FORdocINWITHIN(@@collection,@latitude,@longitude,@radius,@distanceAttributeName)

RETURNdoc

Examples

Tofindalldocumentswithinaradiusof2000kmuse:


........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});}}

arangosh>db.geo.within(0,0,2000*1000).distance().toArray();

showexecutionresultsensuresthatageoindexexistscollection.ensureIndex({type:"geo",fields:["location"]})

GeoIndexes

120

SinceArangoDB2.5,thismethodisanaliasforensureGeoIndexsincegeoindexesarealwayssparse,meaningthatdocumentsthatdonotcontaintheindexattributesorhavenon-numericvaluesintheindexattributeswillnotbeindexed.ensureGeoConstraintisdeprecatedandensureGeoIndexshouldbeusedinstead.

Theindexdoesnotprovideauniqueoptionbecauseofitslimitedusability.Itwouldpreventidenticalcoordinatesfrombeinginsertedonly,butevenaslightlydifferentlocation(like1inchor1cmoff)wouldbeuniqueagainandnotconsideredaduplicate,althoughitprobablyshould.Thedesiredthresholdfordetectingduplicatesmayvaryforeveryproject(includinghowtocalculatethedistanceeven)andneedstobeimplementedontheapplicationlayerasneeded.YoucanwriteaFoxxserviceforthispurposeandmakeuseoftheAQLgeofunctionstofindnearbycoordinatessupportedbyageoindex.

GeoIndexes

121


IntroductiontoVertexCentricIndexes

InArangoDBtherearespecialindicesdesignedtospeedupgraphoperations,especiallyifthegraphcontainssupernodes(verticesthathaveanexceptionallyhighamountofconnectededges).Theseindicesarecalledvertexcentricindexesandcanbeusedinadditiontotheexistingedgeindex.

Motivation

Theideaofthisindexistoindexacombinationofavertex,thedirectionandanyarbitrarysetofotherattributesontheedges.Totakeanexample,ifwehaveanattributecalledtypeontheedges,wecanuseanoutboundvertex-centricindexonthisattributetofindalledgesattachedtoavertexwithagiventype.Thefollowingqueryexamplecouldbenefitfromsuchanindex:

FORv,e,pIN3..5OUTBOUND@startGRAPH@graphName

FILTERp.edges[*].typeALL=="friend"

RETURNv

Usingthebuilt-inedge-indexArangoDBcanfindthelistofalledgesattachedtothevertexfast,butstillithastowalkthroughthislistandcheckifallofthemhavetheattributetype=="friend".Usingavertex-centricindexwouldallowArangoDBtofindalledgesforthevertexhavingtheattributetype=="friend"inthesametimeandcansavetheiterationtoverifythecondition.

IndexcreationAvertex-centriccanbeeitherofthefollowingtypes:

HashIndexSkiplistIndexPersistentIndex

Andiscreatedusingtheircreationoperations.Howeverinthelistoffieldsusedtocreatetheindexwehavetoincludeeither_fromor_to.Letusagainexplainthisbyanexample.Assumewewanttocreateanhash-basedoutboundvertex-centricindexontheattributetype.Thiscanbecreatedwiththefollowingway:

arangosh>db.collection.ensureIndex({type:"hash",fields:["_from","type"]})

showexecutionresultsAlloptionsthataresupportedbytherespectiveindexesaresupportedbythevertex-centricindexaswell.

Indexusage

TheAQLoptimizercandecidetouseavertex-centricwheneversuitable,howeveritisnotguaranteedthatthisindexisused,theoptimizermayestimatethatanotherindexisassumedtobebetter.Theoptimizerwillconsiderthistypeofindexesonexplicitfilteringof_fromrespectively_to:

FORedgeINcollection

FILTERedge._from=="vertices/123456"ANDedge.type=="friend"

RETURNedge

andduringpatternmatchingqueries:

FORv,e,pIN3..5OUTBOUND@startGRAPH@graphName

FILTERp.edges[*].typeALL=="friend"

RETURNv


122


123

ArangoDBGraphs

FirstStepswithGraphs

AGraphconsistsofverticesandedges.Edgesarestoredasdocumentsinedgecollections.Avertexcanbeadocumentofadocumentcollectionorofanedgecollection(soedgescanbeusedasvertices).Whichcollectionsareusedwithinanamedgraphisdefinedviaedgedefinitions.Anamedgraphcancontainmorethanoneedgedefinition,atleastoneisneeded.Graphsallowyoutostructureyourmodelsinlinewithyourdomainandgroupthemlogicallyincollectionsandgivingyouthepowertoquerytheminthesamegraphqueries.

Newtographs?Takeourfreegraphcourseforfreshersandgetfromzeroknowledgetoadvancedquerytechniques.

Comingfromarelationalbackground-what'sagraph?

InSQLyoucommonlyhavetheconstructofarelationtabletostoren:mrelationsbetweentwodatatables.Anedgecollectionissomewhatsimilartotheserelationtables;vertexcollectionsresemblethedatatableswiththeobjectstoconnect.WhilesimplegraphquerieswithfixednumberofhopsviatherelationtablemaybedoableinSQLwithseveralnestedjoins,graphdatabasescanhandleanarbitrarynumberofthesehopsoveredgecollections-thisiscalledtraversal.Alsoedgesinoneedgecollectionmaypointtoseveralvertexcollections.Itscommontohaveattributesattachedtoedges,i.e.alabelnamingthisinterconnection.Edgeshaveadirection,withtheirrelations_fromand_topointingfromonedocumenttoanotherdocumentstoredinvertexcollections.Inqueriesyoucandefineinwhichdirectionstheedgerelationsmaybefollowed(OUTBOUND:_from→_to,INBOUND:_from←_to,ANY:_from↔_to).

NamedGraphs

NamedgraphsarecompletelymanagedbyArangoDB,andthusalsovisibleinthewebinterface.TheyusethefullspectrumofArangoDB'sgraphfeatures.Youmayaccessthemviaseveralinterfaces.

AQLGraphOperationswithseveralflavors:AQLTraversalsonbothnamedandanonymousgraphsAQLShortestPathonbothnamedandanonymousgraph

JavaScriptGeneralGraphimplementation,asyoumayuseitinFoxxServicesGraphManagement;creating&manipualatinggraphdefinitions;inserting,updatinganddeletingverticesandedgesintographsGraphFunctionsforworkingwithedgesandvertices,toanalyzethemandtheirrelations

JavaScriptSmartGraphimplementation,forscalablegraphsSmartGraphManagement;creating&manipualatingSmartGraphdefinitions;DifferencestoGeneralGraph

RESTfulGeneralGraphinterfaceusedtoimplementgraphmanagementinclientdrivers

Manipulatingcollectionsofnamedgraphswithregulardocumentfunctions

Theunderlyingcollectionsofthenamedgraphsarestillaccessibleusingthestandardmethodsforcollections.Howeverthegraphmoduleaddsanadditionallayerontopofthesecollectionsgivingyouthefollowingguarantees:

AllmodificationsareexecutedtransactionalIfyoudeleteavertexalledgeswillbedeleted,youwillneverhavelooseendsIfyouinsertanedgeitischeckediftheedgematchestheedgedefinitions,youredgecollectionswillonlycontainvalidedges

TheseguaranteesarelostifyouaccessthecollectionsinanyotherwaythanthegraphmoduleorAQL,soifyoudeletedocumentsfromyourvertexcollectionsdirectly,theedgespointingtothemwillberemaininplace.

Anonymousgraphs

Sometimesyoumaynotneedallthepowersofnamedgraphs,butsomeofitsbitsmaybevaluabletoyou.YoumayuseanonymousgraphsinthetraversalsandintheWorkingwithEdgeschapter.Anonymousgraphsdon'thaveedgedefinitionsdescribingwhichvertexcollectionisconnectedbywhichedgecollection.Thegraphmodelhastobemaintainedintheclientsidecode.Thisgivesyoumorefreedomthanthestrictnamedgraphs.

Graphs

124

https://www.arangodb.com/arangodb-graph-course/

AQLGraphOperationsareavailableforboth,namedandanonymousgraphs:AQLTraversalsAQLShortestPath

Whentochooseanonymousornamedgraphs?

Asnotedabove,namedgraphsensuregraphintegrity,bothwheninsertingorremovingedgesorvertices.Soyouwon'tencounterdanglingedges,evenifyouusethesamevertexcollectioninseveralnamedgraphs.Thisinvolvesmoreoperationsinsidethedatabasewhichcomeatacost.Thereforeanonymousgraphsmaybefasterinmanyoperations.Sothisquestionmaybenarroweddownto:'CanIaffordtheadditionaleffortordoIneedthewarrantyforintegrity?'.

Multipleedgecollectionsvs.FILTERsonedgedocumentattributes

Ifyouwanttoonlytraverseedgesofaspecifictype,therearetwowaystoachievethis.Thefirstwouldbeanattributeintheedgedocument-i.e.type,whereyouspecifyadifferentiatorfortheedge-i.e."friends","family","married"or"workmates",soyoucanlaterFILTERe.type="friends"ifyouonlywanttofollowthefriendedges.

Anotherway,whichmaybemoreefficientinsomecases,istousedifferentedgecollectionsfordifferenttypesofedges,soyouhavefriend_edges,family_edges,married_edgesandworkmate_edgesascollectionnames.Youcanthenconfigureseveralnamedgraphsincludingasubsetoftheavailableedgeandvertexcollections-oryouuseanonymousgraphqueries,whereyouspecifyalistofedgecollectionstotakeintoaccountinthatquery.Toonlyfollowfriendedges,youwouldspecifyfriend_edgesassoleedgecollection.

Bothapproacheshaveadvantagesanddisadvantages.FILTERoperationsonedgeattributeswilldocomparisonsoneachtraversededge,whichmaybecomeCPU-intense.Whennotfindingtheedgesinthefirstplacebecauseofthecollectioncontainingthemisnottraversedatall,therewillneverbeareasontoactualycheckfortheirtypeattributewithFILTER.

Themultipleedgecollectionsapproachislimitedbythenumberofcollectionsthatcanbeusedsimultaneouslyinonequery.EverycollectionusedinaqueryrequiressomeresourcesinsideofArangoDBandthenumberisthereforelimitedtocaptheresourcerequirements.Youmayalsohaveconstraintsonotheredgeattributes,suchasahashindexwithauniqueconstraint,whichrequiresthedocumentstobeinasinglecollectionfortheuniquenessguarantee,anditmaythusnotbepossibletostorethedifferenttypesofedgesinmultipleedgecollections.

So,ifyouredgeshaveaboutadozendifferenttypes,it'sokaytochoosethecollectionapproach,otherwisetheFILTERapproachispreferred.YoucanstilluseFILTERoperationsonedgesofcourse.YoucangetridofaFILTERonthetypewiththeformerapproach,everythingelsecanstaythesame.

WhichpartofmydataisanEdgeandwhichaVertex?

Themainobjectsinyourdatamodel,suchasusers,groupsorarticles,areusuallyconsideredtobevertices.Foreachtypeofobject,adocumentcollection(alsocalledvertexcollection)shouldstoretheindividualentities.Entitiescanbeconnectedbyedgestoexpressandclassifyrelationsbetweenvertices.Itoftenmakessensetohaveanedgecollectionperrelationtype.

ArangoDBdoesnotrequireyoutostoreyourdataingraphstructureswithedgesandvertices,youcanalsodecidetoembedattributessuchaswhichgroupsauserispartof,or_idsofdocumentsinanotherdocumentinsteadofconnectingthedocumentswithedges.Itcanbeameaningfulperformanceoptimizationfor1:nrelationships,ifyourdataisnotfocusedonrelationsandyoudon'tneedgraphtraversalwithvaryingdepth.Itusuallymeanstointroducesomeredundancyandpossiblyinconsistenciesifyouembeddata,butitcanbeanacceptabletradeoff.

Vertices

Let'ssaywehavetwovertexcollections,UsersandGroups.DocumentsintheGroupscollectioncontaintheattributesoftheGroup,i.e.whenitwasfounded,itssubject,aniconURLandsoon.Usersdocumentscontainthedataspecifictoauser-likeallnames,birthdays,AvatarURLs,hobbies...

Edges

Wecanuseanedgecollectiontostorerelationsbetweenusersandgroups.Sincemultipleusersmaybeinanarbitrarynumberofgroups,thisisanm:nrelation.TheedgecollectioncanbecalledUsersInGroupswithi.e.oneedgewith_frompointingtoUsers/Johnand_topointingtoGroups/BowlingGroupHappyPin.ThismakestheuserJohnamemberofthegroupBowlingGroupHappyPin.

Graphs

125

Attributesofthisrelationmaycontainqualifierstothisrelation,likethepermissionsofJohninthisgroup,thedatewhenhejoinedthegroupetc.

Soroughlyput,ifyouusedocumentsandtheirattributesinasentence,nounswouldtypicallybevertices,verbsbecometheedges.Youcanseethisintheknowsgraphbelow:

AliceknowsBob,whointermknowsCharlie.

Advantagesofthisapproach

Graphsgiveyoutheadvantageofnotjustbeingabletohaveafixednumberofm:nrelationsinarow,butanarbitrarynumber.Edgescanbetraversedinbothdirections,soit'seasytodetermineallgroupsauserisin,butalsotofindoutwhichmembersacertaingrouphas.Userscouldalsobeinterconnectedtocreateasocialnetwork.

Usingthegraphdatamodel,dealingwithdatathathaslotsofrelationsstaysmanageableandcanbequeriedinveryflexibleways,whereasitwouldcauseheadachetohandleitinarelationaldatabasesystem.

Backupandrestore

Forsureyouwanttohavebackupsofyourgraphdata,youcanuseArangodumptocreatethebackup,andArangorestoretorestoreabackupintoanewArangoDB.Youshouldhowevernotethat:

youneedthesystemcollection_graphsifyoubackupnamedgraphs.youneedtobackupthecompletesetofalledgeandvertexcollectionsyourgraphconsistsof.Partialdump/restoremaynotwork.

Managinggraphs

Bydefaultyoushouldusetheinterfaceyourdriverprovidestomanagegraphs.

Thisisi.e.documentedinGraphs-SectionoftheArangoDBJavadriver.

ExampleGraphs

ArangoDBcomeswithasetofeasilygraspablegraphsthatareusedtodemonstratetheAPIs.Youcanusetheaddsamplestabinthecreategraphwindowinthewebinterface,orloadthemodule@arangodb/graph-examples/example-graphinarangoshanduseittocreateinstancesofthesegraphsinyourArangoDB.Onceyou'vecreatedthem,youcaninspecttheminthewebinterface-whichwasusedtocreatethepicturesbelow.

Youcaneasilylookintotheinnardsofthisscriptforreferenceabouthowtomanagegraphsprogramatically.

TheKnows_Graph

Graphs

126

https://github.com/arangodb/arangodb-java-driver#graphs

https://github.com/arangodb/arangodb/blob/devel/js/common/modules/%40arangodb/graph-examples/example-graph.js

Asetofpersonsknowingeachother:

Theknowsgraphconsistsofonevertexcollectionpersonsconnectedviaoneedgecollectionknows.ItwillcontainfivepersonsAlice,Bob,Charlie,DaveandEve.Wewillhavethefollowingdirectedrelations:

AliceknowsBobBobknowsCharlieBobknowsDaveEveknowsAliceEveknowsBob

Thisishowwecreateit,inspectitsverticesandedges,anddropitagain:

arangosh>varexamples=require("@arangodb/graph-examples/example-graph.js");

arangosh>varg=examples.loadGraph("knows_graph");

arangosh>db.persons.toArray()

arangosh>db.knows.toArray();

arangosh>examples.dropGraph("knows_graph");


TheSocialGraph

Asetofpersonsandtheirrelations:

Graphs

127

Thisexamplehasfemaleandmalepersonsasverticesintwovertexcollections-femaleandmale.Theedgesaretheirconnectionsintherelationedgecollection.Thisishowwecreateit,inspectitsverticesandedges,anddropitagain:


arangosh>vargraph=examples.loadGraph("social");

arangosh>db.female.toArray()

arangosh>db.male.toArray()

arangosh>db.relation.toArray()

arangosh>examples.dropGraph("social");


TheCityGraph

Asetofeuropeancities,andtheirfictionaltravelingdistancesasconnections:

Graphs

128

Theexamplehasthecitiesasverticesinseveralvertexcollections-germanCityandfrenchCity.Theedgesaretheirinterconnectionsinseveraledgecollectionsfrench/german/internationalHighway.Thisishowwecreateit,inspectitsedgesandvertices,anddropitagain:


arangosh>varg=examples.loadGraph("routeplanner");

arangosh>db.frenchCity.toArray();

arangosh>db.germanCity.toArray();

arangosh>db.germanHighway.toArray();

arangosh>db.frenchHighway.toArray();

arangosh>db.internationalHighway.toArray();

arangosh>examples.dropGraph("routeplanner");


TheTraversalGraph

Thisgraphwasdesignedtodemonstratefiltersintraversals.Ithassomelabelstofilteronit.

Graphs

129

Theexamplehasallitsverticesinthecirclescollection,andanedgesedgecollectiontoconnectthem.Circleshaveuniquenumericlabels.Edgeshavetwobooleanattributes(theFalsealwaysbeingfalse,theTruthalwaysbeingtrue)andalabelsortingB-Dtotheleftside,G-Ktotherightside.LeftandrightsidesplitintoPaths-atBandGwhichareeachdirectneighboursoftheroot-nodeA.StartingfromAthegraphhasadepthof3onallitspaths.


arangosh>varg=examples.loadGraph("traversalGraph");

arangosh>db.circles.toArray();

arangosh>db.edges.toArray();

arangosh>examples.dropGraph("traversalGraph");


TheWorldGraph

Theworldcountrygraphstructuresitsnodeslikethat:world→continent→country→capital.Insomecasesedgedirectionsaren'tforward(thereforeitwillbedisplayeddisjunctinthegraphviewer).Ithastwowaysofcreatingit.Oneusingthenamedgraphutilities(worldCountry),onewithout(worldCountryUnManaged).Itisusedtodemonstraterawtraversaloperations.


arangosh>varg=examples.loadGraph("worldCountry");

arangosh>db.worldVertices.toArray();

arangosh>db.worldEdges.toArray();

arangosh>examples.dropGraph("worldCountry");

arangosh>varg=examples.loadGraph("worldCountryUnManaged");

arangosh>examples.dropGraph("worldCountryUnManaged");


Graphs

130

Cookbookexamples

TheabovereferencedchaptersdescribethevariousAPIsofArangoDBsgraphenginewithsmallexamples.Ourcookbookhassomemorereallifeexamples:

TraversingagraphinfulldepthUsinganexamplevertexwiththejavadriverRetrievingdocumentsfromArangoDBwithoutknowingthestructureUsingacustomvisitorfromnode.jsAQLExampleQueriesonanActorsandMoviesDatabase

Highervolumegraphexamples

Alloftheaboveexamplesarerathersmallsotheyareeasiertocomprehendandcandemonstratethewaythefunctionalityworks.Therearehoweverseveraldatasetsfreelyavailableonthewebthatarealotbigger.Wecollectedsomeofthemwithimportscriptssoyoumayplayaroundwiththem.AnotherhugegraphisthePokecsocialnetworkfromSlovakiathatweusedforperformancetestingonseveraldatabases;Youwillfindimportingscriptsetc.inthisblogpost.

Graphs

131

https://github.com/arangodb/example-datasets

https://snap.stanford.edu/data/soc-pokec.html

https://www.arangodb.com/2015/06/multi-model-benchmark/

GraphsThischapterdescribesthegeneral-graphmodule.Itallowsyoutodefineagraphthatisspreadacrossseveraledgeanddocumentcollections.Thisallowsyoutostructureyourmodelsinlinewithyourdomainandgroupthemlogicallyincollectionsgivingyouthepowertoquerytheminthesamegraphqueries.Thereisnoneedtoincludethereferencedcollectionswithinthequery,thismodulewillhandleitforyou.

ThreeStepstocreateagraph

Createagraph

arangosh>vargraph_module=require("@arangodb/general-graph");

arangosh>vargraph=graph_module._create("myGraph");

arangosh>graph;

{[Graph]

}

Addsomevertexcollections

arangosh>graph._addVertexCollection("shop");

arangosh>graph._addVertexCollection("customer");

arangosh>graph._addVertexCollection("pet");

arangosh>graph;

showexecutionresultsDefinerelationsontheGraph

arangosh>varrel=graph_module._relation("isCustomer",["shop"],["customer"]);

arangosh>graph._extendEdgeDefinitions(rel);

arangosh>graph;


GeneralGraphs

132

GraphManagementThischapterdescribesthejavascriptinterfaceforcreatingandmodifyingnamedgraphs.Inordertocreateanonemptygraphthefunctionalitytocreateedgedefinitionshastobeintroducedfirst:

EdgeDefinitions

Anedgedefinitionisalwaysadirectedrelationofagraph.Eachgraphcanhavearbitrarymanyrelationsdefinedwithintheedgedefinitionsarray.

Initializethelist

Createalistofedgedefinitionstoconstructagraph.

graph_module._edgeDefinitions(relation1,relation2,...,relationN)

Thelistofedgedefinitionsofagraphcanbemanagedbythegraphmoduleitself.Thisfunctionistheentrypointforthemanagementandwillreturnthecorrectlist.

Parameters

relationX(optional)Anobjectrepresentingadefinitionofonerelationinthegraph

Examples


arangosh>directed_relation=graph_module._relation("lives_in","user","city");

arangosh>undirected_relation=graph_module._relation("knows","user","user");

arangosh>edgedefinitions=graph_module._edgeDefinitions(directed_relation,

undirected_relation);


Extendthelist

Extendthelistofedgedefinitionstoconstructagraph.

graph_module._extendEdgeDefinitions(edgeDefinitions,relation1,relation2,...,relationN)

Inordertoaddmoreedgedefinitionstothegraphbeforecreatingthisfunctioncanbeusedtoaddmoredefinitionstotheinitiallist.

Parameters

edgeDefinitions(required)Alistofrelationdefinitionobjects.relationX(required)Anobjectrepresentingadefinitionofonerelationinthegraph

Examples


arangosh>directed_relation=graph_module._relation("lives_in","user","city");

arangosh>undirected_relation=graph_module._relation("knows","user","user");

arangosh>edgedefinitions=graph_module._edgeDefinitions(directed_relation);

arangosh>edgedefinitions=graph_module._extendEdgeDefinitions(undirected_relation);


Relation

Defineadirectedrelation.

GraphManagement

133

graph_module._relation(relationName,fromVertexCollections,toVertexCollections)

TherelationNamedefinesthenameofthisrelationandreferencestotheunderlyingedgecollection.ThefromVertexCollectionsisanArrayofdocumentcollectionsholdingthestartvertices.ThetoVertexCollectionsisanArrayofdocumentcollectionsholdingthetargetvertices.RelationsareonlyallowedinthedirectionfromanycollectioninfromVertexCollectionstoanycollectionintoVertexCollections.

Parameters

relationName(required)Thenameoftheedgecollectionwheretheedgesshouldbestored.Willbecreatedifitdoesnotyetexist.fromVertexCollections(required)Oneoralistofcollectionnames.Sourceverticesfortheedgeshavetobestoredinthesecollections.Collectionswillbecreatediftheydonotexist.toVertexCollections(required)Oneoralistofcollectionnames.Targetverticesfortheedgeshavetobestoredinthesecollections.Collectionswillbecreatediftheydonotexist.

Examples


arangosh>graph_module._relation("has_bought",["Customer","Company"],["Groceries",

"Electronics"]);



arangosh>graph_module._relation("has_bought","Customer","Product");


CreateagraphAfterhavingintroducededgedefinitionsagraphcanbecreated.

Createagraph

graph_module._create(graphName,edgeDefinitions,orphanCollections)

Thecreationofagraphrequiresthenameofthegraphandadefinitionofitsedges.

Foreverytypeofedgedefinitionaconveniencemethodexiststhatcanbeusedtocreateagraph.Optionallyalistofvertexcollectionscanbeadded,whicharenotusedinanyedgedefinition.Thesecollectionsarereferredtoasorphancollectionswithinthischapter.Allcollectionsusedwithinthecreationprocessarecreatediftheydonotexist.

Parameters

graphName(required)UniqueidentifierofthegraphedgeDefinitions(optional)ListofrelationdefinitionobjectsorphanCollections(optional)Listofadditionalvertexcollectionnames

Examples

Createanemptygraph,edgedefinitionscanbeaddedatruntime:


arangosh>graph=graph_module._create("myGraph");

{[Graph]

}

Createagraphusinganedgecollectionedgesandasinglevertexcollectionvertices


arangosh>varedgeDefinitions=[{collection:"edges","from":["vertices"],"to":[

"vertices"]}];

GraphManagement

134

arangosh>graph=graph_module._create("myGraph",edgeDefinitions);

{[Graph]

"edges":[ArangoCollection16644,"edges"(typeedge,statusloaded)],

"vertices":[ArangoCollection16639,"vertices"(typedocument,statusloaded)]

}

Createagraphwithedgedefinitionsandorphancollections:


arangosh>graph=graph_module._create("myGraph",

........>[graph_module._relation("myRelation",["male","female"],["male","female"])],

["sessions"]);


CompleteExampletocreateagraph

ExampleCall:


arangosh>varedgeDefinitions=graph_module._edgeDefinitions();

arangosh>graph_module._extendEdgeDefinitions(edgeDefinitions,

graph_module._relation("friend_of","Customer","Customer"));

arangosh>graph_module._extendEdgeDefinitions(

........>edgeDefinitions,graph_module._relation(

........>"has_bought",["Customer","Company"],["Groceries","Electronics"]));

arangosh>graph_module._create("myStore",edgeDefinitions);

showexecutionresultsalternativecall:


arangosh>varedgeDefinitions=graph_module._edgeDefinitions(

........>graph_module._relation("friend_of",["Customer"],["Customer"]),

graph_module._relation(

........>"has_bought",["Customer","Company"],["Groceries","Electronics"]));

arangosh>graph_module._create("myStore",edgeDefinitions);


Listavailablegraphs

Listallgraphs.

graph_module._list()

Listsallgraphnamesstoredinthisdatabase.

Examples


arangosh>graph_module._list();

[]

Loadagraph

GraphManagement

135

Getagraph

graph_module._graph(graphName)

Agraphcanberetrievedbyitsname.

Parameters

graphName(required)Uniqueidentifierofthegraph

Examples

Getagraph:


arangosh>graph=graph_module._graph("social");


Removeagraph

Removeagraph

graph_module._drop(graphName,dropCollections)

Agraphcanbedroppedbyitsname.Thiscandropallcollectionscontainedinthegraphaslongastheyarenotusedwithinothergraphs.Todropthecollectionsonlybelongingtothisgraph,theoptionalparameterdrop-collectionshastobesettotrue.

Parameters

graphName(required)UniqueidentifierofthegraphdropCollections(optional)Defineifcollectionsshouldbedropped(default:false)

Examples

Dropagraphandkeepcollections:


arangosh>graph_module._drop("social");

true

arangosh>db._collection("female");

[ArangoCollection16731,"female"(typedocument,statusloaded)]

arangosh>db._collection("male");

[ArangoCollection16734,"male"(typedocument,statusloaded)]

arangosh>db._collection("relation");

[ArangoCollection16737,"relation"(typeedge,statusloaded)]


arangosh>graph_module._drop("social",true);

true

arangosh>db._collection("female");

null

arangosh>db._collection("male");

null

arangosh>db._collection("relation");

null

ModifyagraphdefinitionduringruntimeAfteryouhavecreatedangraphitsdefinitionisnotimmutable.Youcanstilladd,deleteormodifyedgedefinitionsandvertexcollections.

GraphManagement

136

Extendtheedgedefinitions

Addanotheredgedefinitiontothegraph

graph._extendEdgeDefinitions(edgeDefinition)

Extendstheedgedefinitionsofagraph.Ifanorphancollectionisusedinthisedgedefinition,itwillberemovedfromtheorphanage.Iftheedgecollectionoftheedgedefinitiontoaddisalreadyusedinthegraphorusedinadifferentgraphwithdifferentfromand/ortocollectionsanerroristhrown.

Parameters

edgeDefinition(required)Therelationdefinitiontoextendthegraph

Examples

arangosh>vargraph_module=require("@arangodb/general-graph")

arangosh>vared1=graph_module._relation("myEC1",["myVC1"],["myVC2"]);


arangosh>vargraph=graph_module._create("myGraph",[ed1]);

arangosh>graph._extendEdgeDefinitions(ed2);

Modifyanedgedefinition

Modifyanrelationdefinition

graph_module._editEdgeDefinition(edgeDefinition)

Editsonerelationdefinitionofagraph.Theedgedefinitionusedasargumentwillreplacetheexistingedgedefinitionofthegraphwhichhasthesamecollection.VertexCollectionsofthereplacededgedefinitionthatarenotusedinthenewdefinitionwilltransformtoanorphan.Orphansthatareusedinthisnewedgedefinitionwillbedeletedfromthelistoforphans.Othergraphswiththesameedgedefinitionwillbemodified,too.

Parameters

edgeDefinition(required)Theedgedefinitiontoreplacetheexistingedgedefinitionwiththesameattributecollection.

Examples


arangosh>varoriginal=graph_module._relation("myEC1",["myVC1"],["myVC2"]);

arangosh>varmodified=graph_module._relation("myEC1",["myVC2"],["myVC3"]);

arangosh>vargraph=graph_module._create("myGraph",[original]);

arangosh>graph._editEdgeDefinitions(modified);

Deleteanedgedefinition

Deleteonerelationdefinition

graph_module._deleteEdgeDefinition(edgeCollectionName,dropCollection)

Deletesarelationdefinitiondefinedbytheedgecollectionofagraph.Ifthecollectionsdefinedintheedgedefinition(collection,from,to)arenotusedinanotheredgedefinitionofthegraph,theywillbemovedtotheorphanage.

Parameters

edgeCollectionName(required)Nameofedgecollectionintherelationdefinition.dropCollection(optional)Defineiftheedgecollectionshouldbedropped.Defaultfalse.

Examples

Removeanedgedefinitionbutkeeptheedgecollection:


GraphManagement

137



arangosh>vargraph=graph_module._create("myGraph",[ed1,ed2]);

arangosh>graph._deleteEdgeDefinition("myEC1");

arangosh>db._collection("myEC1");

[ArangoCollection22374,"myEC1"(typeedge,statusloaded)]

Removeanedgedefinitionanddroptheedgecollection:




arangosh>vargraph=graph_module._create("myGraph",[ed1,ed2]);

arangosh>graph._deleteEdgeDefinition("myEC1",true);

arangosh>db._collection("myEC1");

null

ExtendvertexCollections

Eachgraphcanhaveanarbitraryamountofvertexcollections,whicharenotpartofanyedgedefinitionofthegraph.Thesecollectionsarecalledorphancollections.Ifthegraphisextendedwithanedgedefinitionusingoneoftheorphans,itwillberemovedfromthesetoforphancollectionautomatically.

Addavertexcollection

Addavertexcollectiontothegraph

graph._addVertexCollection(vertexCollectionName,createCollection)

Addsavertexcollectiontothesetoforphancollectionsofthegraph.Ifthecollectiondoesnotexist,itwillbecreated.Ifitisalreadyusedbyanyedgedefinitionofthegraph,anerrorwillbethrown.

Parameters

vertexCollectionName(required)Nameofvertexcollection.createCollection(optional)Iftruethecollectionwillbecreatedifitdoesnotexist.Default:true.

Examples




arangosh>graph._addVertexCollection("myVC3",true);

Gettheorphanedcollections

Getallorphancollections

graph._orphanCollections()

Returnsallvertexcollectionsofthegraphthatarenotusedinanyedgedefinition.

Examples





arangosh>graph._orphanCollections();

GraphManagement

138

[

"myVC3"

]

Removeavertexcollection

Removeavertexcollectionfromthegraph

graph._removeVertexCollection(vertexCollectionName,dropCollection)

Removesavertexcollectionfromthegraph.Onlycollectionsnotusedinanyrelationdefinitioncanberemoved.Optionallythecollectioncanbedeleted,ifitisnotusedinanyothergraph.

Parameters

vertexCollectionName(required)Nameofvertexcollection.dropCollection(optional)Iftruethecollectionwillbedroppedifitisnotusedinanyothergraph.Default:false.

Examples







arangosh>graph._removeVertexCollection("myVC3");



ManiuplatingVertices

Saveavertex

CreateanewvertexinvertexCollectionName

graph.vertexCollectionName.save(data)

Parameters

data(required)JSONdataofvertex.

Examples



arangosh>graph.male.save({name:"Floyd",_key:"floyd"});


Replaceavertex

ReplacesthedataofavertexincollectionvertexCollectionName

graph.vertexCollectionName.replace(vertexId,data,options)

Parameters

vertexId(required)_idattributeofthevertexdata(required)JSONdataofvertex.

GraphManagement

139

options(optional)Seecollectiondocumentation

Examples



arangosh>graph.male.save({neym:"Jon",_key:"john"});

arangosh>graph.male.replace("male/john",{name:"John"});


Updateavertex

UpdatesthedataofavertexincollectionvertexCollectionName

graph.vertexCollectionName.update(vertexId,data,options)

Parameters

vertexId(required)_idattributeofthevertexdata(required)JSONdataofvertex.options(optional)Seecollectiondocumentation

Examples



arangosh>graph.female.save({name:"Lynda",_key:"linda"});

arangosh>graph.female.update("female/linda",{name:"Linda",_key:"linda"});


Removeavertex

RemovesavertexincollectionvertexCollectionName

graph.vertexCollectionName.remove(vertexId,options)

Additionallyremovesallingoingandoutgoingedgesofthevertexrecursively(seeedgeremove).

Parameters

vertexId(required)_idattributeofthevertexoptions(optional)Seecollectiondocumentation

Examples



arangosh>graph.male.save({name:"Kermit",_key:"kermit"});

arangosh>db._exists("male/kermit")

arangosh>graph.male.remove("male/kermit")

arangosh>db._exists("male/kermit")


ManipulatingEdges

Saveanewedge

GraphManagement

140

CreatesanedgefromvertexfromtovertextoincollectionedgeCollectionName

graph.edgeCollectionName.save(from,to,data,options)

Parameters

from(required)_idattributeofthesourcevertexto(required)_idattributeofthetargetvertexdata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation

Examples



arangosh>graph.relation.save("male/bob","female/alice",{type:"married",_key:

"bobAndAlice"});

showexecutionresultsIfthecollectionsoffromandtoarenotdefinedinanedgedefinitionofthegraph,theedgewillnotbestored.



arangosh>graph.relation.save(

........>"relation/aliceAndBob",

........>"female/alice",

........>{type:"married",_key:"bobAndAlice"});

[ArangoError1906:invalidedgebetweenrelation/aliceAndBobandfemale/alice.Doesn't

conformtoanyedgedefinition]

Replaceanedge

ReplacesthedataofanedgeincollectionedgeCollectionName.Notethat_fromand_toaremandatory.

graph.edgeCollectionName.replace(edgeId,data,options)

Parameters

edgeId(required)_idattributeoftheedgedata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation

Examples



arangosh>graph.relation.save("female/alice","female/diana",{typo:"nose",_key:

"aliceAndDiana"});

arangosh>graph.relation.replace("relation/aliceAndDiana",{type:"knows",_from:

"female/alice",_to:"female/diana"});


Updateanedge

UpdatesthedataofanedgeincollectionedgeCollectionName

graph.edgeCollectionName.update(edgeId,data,options)

Parameters

GraphManagement

141

edgeId(required)_idattributeoftheedgedata(required)JSONdataoftheedgeoptions(optional)Seecollectiondocumentation

Examples



arangosh>graph.relation.save("female/alice","female/diana",{type:"knows",_key:

"aliceAndDiana"});

arangosh>graph.relation.update("relation/aliceAndDiana",{type:"quarreled",_key:

"aliceAndDiana"});


Removeanedge

RemovesanedgeincollectionedgeCollectionName

graph.edgeCollectionName.remove(edgeId,options)

Ifthisedgeisusedasavertexbyanotheredge,theotheredgewillberemoved(recursively).

Parameters

edgeId(required)_idattributeoftheedgeoptions(optional)Seecollectiondocumentation

Examples



arangosh>graph.relation.save("female/alice","female/diana",{_key:"aliceAndDiana"});

arangosh>db._exists("relation/aliceAndDiana")

arangosh>graph.relation.remove("relation/aliceAndDiana")

arangosh>db._exists("relation/aliceAndDiana")


Connectedges

Getallconnectingedgesbetween2groupsofverticesdefinedbytheexamples

graph._getConnectingEdges(vertexExample,vertexExample2,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.

Parameters

vertexExample1(optional)SeeDefinitionofexamplesvertexExample2(optional)SeeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

edgeExamples:Filtertheedges,seeDefinitionofexamplesedgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.vertex1CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.vertex2CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.

Examples

Arouteplannerexample,allconnectingedgesbetweencapitals.

GraphManagement

142


arangosh>vargraph=examples.loadGraph("routeplanner");

arangosh>graph._getConnectingEdges({isCapital:true},{isCapital:true});

[]

GraphManagement

143

GraphFunctionsThischapterdescribesvariousfunctionsonagraph.Alotoftheseacceptavertex(oredge)exampleasparameterasdefinedinthenextsection.

ExampleswillexplaintheAPIonthethecitygraph:

DefinitionofexamplesFormanyofthefollowingfunctionsexamplescanbepassedinasaparameter.Examplesareusedtofiltertheresultsetforobjectsthatmatchtheconditions.Theseexamplescanhavethefollowingvalues:

null,thereisnomatchingexecutedallfoundresultsarevalid.Astring,onlyresultsarereturned,which_idequalthevalueofthestringAnexampleobject,definingasetofattributes.Onlyresultshavingtheseattributesarematched.Alistcontainingexampleobjectsand/orstrings.Allresultsmatchingatleastoneoftheelementsinthelistarereturned.

Getverticesfromedges.

Getvertexfromofanedge

Getthesourcevertexofanedge

graph._fromVertex(edgeId)

Returnsthevertexdefinedwiththeattribute_fromoftheedgewithedgeIdasits_id.

Parameters

edgeId(required)_idattributeoftheedge

Examples

GraphFunctions

144



arangosh>varany=require("@arangodb").db.relation.any();

arangosh>graph._fromVertex("relation/"+any._key);


Getvertextoofanedge

Getthetargetvertexofanedge

graph._toVertex(edgeId)

Returnsthevertexdefinedwiththeattribute_tooftheedgewithedgeIdasits_id.

Parameters

edgeId(required)_idattributeoftheedge

Examples



arangosh>varany=require("@arangodb").db.relation.any();

arangosh>graph._toVertex("relation/"+any._key);


_neighbors

Getallneighborsoftheverticesdefinedbytheexample

graph._neighbors(vertexExample,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertexExample.ThecomplexityofthismethodisO(n*m^x)withnbeingtheverticesdefinedbytheparametervertexExamplex,mtheaverageamountofneighborsandxthemaximaldepths.HencethedefaultcallwouldhaveacomplexityofO(n*m);

Parameters

vertexExample(optional)SeeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeExamples:Filtertheedges,seeDefinitionofexamplesneighborExamples:Filtertheneighborvertices,seeDefinitionofexamplesedgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.vertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredontheintermediatevertexsteps.minDepth:Definestheminimalnumberofintermediatestepstoneighbors(defaultis1).maxDepth:Definesthemaximalnumberofintermediatestepstoneighbors(defaultis1).

Examples

Arouteplannerexample,allneighborsofcapitals.



arangosh>graph._neighbors({isCapital:true});

showexecutionresultsArouteplannerexample,alloutboundneighborsofHamburg.

GraphFunctions

145



arangosh>graph._neighbors('germanCity/Hamburg',{direction:'outbound',maxDepth:2});


_commonNeighbors

Getallcommonneighborsoftheverticesdefinedbytheexamples.

graph._commonNeighbors(vertex1Example,vertex2Examples,optionsVertex1,optionsVertex2)

Thisfunctionreturnstheintersectionofgraph_module._neighbors(vertex1Example,optionsVertex1)andgraph_module._neighbors(vertex2Example,optionsVertex2).Forparameterdocumentationsee_neighbors.

ThecomplexityofthismethodisO(n*m^x)withnbeingthemaximalamountofverticesdefinedbytheparametersvertexExamples,mtheaverageamountofneighborsandxthemaximaldepths.HencethedefaultcallwouldhaveacomplexityofO(n*m);

Examples

Arouteplannerexample,allcommonneighborsofcapitals.



arangosh>graph._commonNeighbors({isCapital:true},{isCapital:true});

showexecutionresultsArouteplannerexample,allcommonoutboundneighborsofHamburgwithanyotherlocationwhichhaveamaximaldepthof2:



arangosh>graph._commonNeighbors(

........>'germanCity/Hamburg',

........>{},

........>{direction:'outbound',maxDepth:2},

........>{direction:'outbound',maxDepth:2});


_countCommonNeighborsGettheamountofcommonneighborsoftheverticesdefinedbytheexamples.

graph._countCommonNeighbors(vertex1Example,vertex2Examples,optionsVertex1,optionsVertex2)

Similarto_commonNeighborsbutreturnscountinsteadoftheelements.

Examples

Arouteplannerexample,allcommonneighborsofcapitals.



arangosh>varexample={isCapital:true};

arangosh>varoptions={includeData:true};

arangosh>graph._countCommonNeighbors(example,example,options,options);


GraphFunctions

146

Arouteplannerexample,allcommonoutboundneighborsofHamburgwithanyotherlocationwhichhaveamaximaldepthof2:



arangosh>varoptions={direction:'outbound',maxDepth:2,includeData:true};

arangosh>graph._countCommonNeighbors('germanCity/Hamburg',{},options,options);


_commonPropertiesGettheverticesofthegraphthatsharecommonproperties.

graph._commonProperties(vertex1Example,vertex2Examples,options)

Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforvertex1Exampleandvertex2Example.

ThecomplexityofthismethodisO(n)withnbeingthemaximalamountofverticesdefinedbytheparametersvertexExamples.

Parameters

vertex1Examples(optional)Filterthesetofsourcevertices,seeDefinitionofexamples

vertex2Examples(optional)Filterthesetofverticescomparedto,seeDefinitionofexamples

options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:vertex1CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbesearchedforsourcevertices.vertex2CollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbesearchedforcomparevertices.ignoreProperties:Oneoralistofattributenamesofadocumentthatshouldbeignored.

Examples

Arouteplannerexample,alllocationswiththesameproperties:



arangosh>graph._commonProperties({},{});

showexecutionresultsArouteplannerexample,allcitieswhichsharesamepropertiesexceptforpopulation.



arangosh>graph._commonProperties({},{},{ignoreProperties:'population'});


_countCommonPropertiesGettheamountofverticesofthegraphthatsharecommonproperties.

graph._countCommonProperties(vertex1Example,vertex2Examples,options)

Similarto_commonPropertiesbutreturnscountinsteadoftheobjects.

Examples

Arouteplannerexample,alllocationswiththesameproperties:


GraphFunctions

147


arangosh>graph._countCommonProperties({},{});

showexecutionresultsArouteplannerexample,allGermancitieswhichsharesamepropertiesexceptforpopulation.



arangosh>graph._countCommonProperties({},{},{vertex1CollectionRestriction:

'germanCity',

........>vertex2CollectionRestriction:'germanCity',ignoreProperties:'population'});


_paths

The_pathsfunctionreturnsallpathsofagraph.

graph._paths(options)

Thisfunctiondeterminesallavailablepathsinagraph.

ThecomplexityofthismethodisO(n*n*m)withnbeingtheamountofverticesinthegraphandmtheaverageamountofconnectededges;

Parameters

options(optional)Anobjectcontainingoptions,seebelow:direction:Thedirectionoftheedges.Possiblevaluesareany,inboundandoutbound(default).followCycles(optional):Ifsettotruethequeryfollowscyclesinthegraph,defaultisfalse.minLength(optional):Definestheminimallengthapathmusthavetobereturned(defaultis0).maxLength(optional):Definesthemaximallengthapathmusthavetobereturned(defaultis10).

Examples

Returnallpathsofthegraph"social":


arangosh>varg=examples.loadGraph("social");

arangosh>g._paths();

showexecutionresultsReturnallinboundpathsofthegraph"social"withamaximallengthof1andaminimallengthof2:


arangosh>varg=examples.loadGraph("social");

arangosh>g._paths({direction:'inbound',minLength:1,maxLength:2});


_shortestPath

The_shortestPathfunctionreturnsallshortestpathsofagraph.

graph._shortestPath(startVertexExample,endVertexExample,options)

GraphFunctions

148

Thisfunctiondeterminesallshortestpathsinagraph.Thefunctionacceptsanid,anexample,alistofexamplesorevenanemptyexampleasparameterforstartandendvertex.Thelengthofapathisbydefaulttheamountofedgesfromonestartvertextoanendvertex.Theoptionweightallowstheusertodefineanedgeattributerepresentingthelength.

Parameters

startVertexExample(optional)AnexampleforthedesiredstartVertices(seeDefinitionofexamples).endVertexExample(optional)AnexampleforthedesiredendVertices(seeDefinitionofexamples).options(optional)Anobjectcontainingoptions,seebelow:

direction:Thedirectionoftheedgesasastring.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneormultipleedgecollectionnames.Onlyedgesfromthesecollectionswillbeconsideredforthepath.startVertexCollectionRestriction:Oneormultiplevertexcollectionnames.Onlyverticesfromthesecollectionswillbeconsideredasstartvertexofapath.endVertexCollectionRestriction:Oneormultiplevertexcollectionnames.Onlyverticesfromthesecollectionswillbeconsideredasendvertexofapath.weight:Thenameoftheattributeoftheedgescontainingthelengthasastring.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedaslength.IfnodefaultissuppliedthedefaultwouldbepositiveInfinitysothepathcouldnotbecalculated.

Examples

Arouteplannerexample,shortestpathfromallgermantoallfrenchcities:



arangosh>g._shortestPath({},{},{weight:'distance',endVertexCollectionRestriction:

'frenchCity',

........>startVertexCollectionRestriction:'germanCity'});

showexecutionresultsArouteplannerexample,shortestpathfromHamburgandColognetoLyon:



arangosh>g._shortestPath([{_id:'germanCity/Cologne'},{_id:'germanCity/Munich'}],

'frenchCity/Lyon',

........>{weight:'distance'});


_distanceTo

The_distanceTofunctionreturnsallpathsandtheredistancewithinagraph.

graph._distanceTo(startVertexExample,endVertexExample,options)

Thisfunctionisawrapperofgraph._shortestPath.Itdoesnotreturntheactualpathbutonlythedistancebetweentwovertices.

Examples

Arouteplannerexample,shortestdistancefromallgermantoallfrenchcities:



arangosh>g._distanceTo({},{},{weight:'distance',endVertexCollectionRestriction:

'frenchCity',

........>startVertexCollectionRestriction:'germanCity'});

GraphFunctions

149

showexecutionresultsArouteplannerexample,shortestdistancefromHamburgandColognetoLyon:



arangosh>g._distanceTo([{_id:'germanCity/Cologne'},{_id:'germanCity/Munich'}],

'frenchCity/Lyon',

........>{weight:'distance'});


_absoluteEccentricityGettheeccentricityoftheverticesdefinedbytheexamples.

graph._absoluteEccentricity(vertexExample,options)


Parameters

vertexExample(optional)Filterthevertices,seeDefinitionofexamplesoptions(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:

direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.startVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredforsourcevertices.endVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredfortargetvertices.weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheeccentricitycannotbecalculated.

Examples

Arouteplannerexample,theabsoluteeccentricityofalllocations.



arangosh>graph._absoluteEccentricity({});

showexecutionresultsArouteplannerexample,theabsoluteeccentricityofalllocations.Thisconsiderstheactualdistances.



arangosh>graph._absoluteEccentricity({},{weight:'distance'});

showexecutionresultsArouteplannerexample,theabsoluteeccentricityofallcitiesregardingonlyoutboundpaths.



arangosh>graph._absoluteEccentricity({},{startVertexCollectionRestriction:

'germanCity',

........>direction:'outbound',weight:'distance'});


GraphFunctions

150

http://en.wikipedia.org/wiki/Distance_%28graph_theory%29

_eccentricity

Getthenormalizedeccentricityoftheverticesdefinedbytheexamples.

graph._eccentricity(vertexExample,options)

Similarto_absoluteEccentricitybutreturnsanormalizedresult.

Examples

Arouteplannerexample,theeccentricityofalllocations.



arangosh>graph._eccentricity();

showexecutionresultsArouteplannerexample,theweightedeccentricity.



arangosh>graph._eccentricity({weight:'distance'});


_absoluteCloseness

Gettheclosenessoftheverticesdefinedbytheexamples.

graph._absoluteCloseness(vertexExample,options)


Parameters


direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).edgeCollectionRestriction:Oneoralistofedge-collectionnamesthatshouldbeconsideredtobeonthepath.startVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredforsourcevertices.endVertexCollectionRestriction:Oneoralistofvertex-collectionnamesthatshouldbeconsideredfortargetvertices.weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheclosenesscannotbecalculated.

Examples

Arouteplannerexample,theabsoluteclosenessofalllocations.



arangosh>graph._absoluteCloseness({});

showexecutionresultsArouteplannerexample,theabsoluteclosenessofalllocations.Thisconsiderstheactualdistances.



GraphFunctions

151

http://en.wikipedia.org/wiki/Distance_%28graph_theory%29

http://en.wikipedia.org/wiki/Centrality#Closeness_centrality

arangosh>graph._absoluteCloseness({},{weight:'distance'});

showexecutionresultsArouteplannerexample,theabsoluteclosenessofallGermanCitiesregardingonlyoutboundpaths.



arangosh>graph._absoluteCloseness({},{startVertexCollectionRestriction:'germanCity',

........>direction:'outbound',weight:'distance'});


_closeness

Getthenormalizedclosenessofgraphsvertices.

graph._closeness(options)

Similarto_absoluteClosenessbutreturnsanormalizedvalue.

Examples

Arouteplannerexample,thenormalizedclosenessofalllocations.



arangosh>graph._closeness();

showexecutionresultsArouteplannerexample,theclosenessofalllocations.Thisconsiderstheactualdistances.



arangosh>graph._closeness({weight:'distance'});

showexecutionresultsArouteplannerexample,theclosenessofallcitiesregardingonlyoutboundpaths.



arangosh>graph._closeness({direction:'outbound',weight:'distance'});


_absoluteBetweennessGetthebetweennessofallverticesinthegraph.

graph._absoluteBetweenness(vertexExample,options)

Parameters


direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthis

GraphFunctions

152

http://en.wikipedia.org/wiki/Centrality#Closeness_centrality

http://en.wikipedia.org/wiki/Betweenness_centrality

defaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencethebetweenesscannotbecalculated.

Examples

Arouteplannerexample,theabsolutebetweennessofalllocations.



arangosh>graph._absoluteBetweenness({});

showexecutionresultsArouteplannerexample,theabsolutebetweennessofalllocations.Thisconsiderstheactualdistances.



arangosh>graph._absoluteBetweenness({weight:'distance'});

{

}

Arouteplannerexample,theabsolutebetweennessofallcitiesregardingonlyoutboundpaths.



arangosh>graph._absoluteBetweenness({direction:'outbound',weight:'distance'});

{

}

_betweenness

Getthenormalizedbetweennessofgraphsvertices.

graph_module._betweenness(options)

Similarto_absoluteBetweenessbutreturnsnormalizedvalues.

Examples

Arouteplannerexample,thebetweennessofalllocations.



arangosh>graph._betweenness();

showexecutionresultsArouteplannerexample,thebetweennessofalllocations.Thisconsiderstheactualdistances.



arangosh>graph._betweenness({weight:'distance'});

showexecutionresultsArouteplannerexample,thebetweennessofallcitiesregardingonlyoutboundpaths.



arangosh>graph._betweenness({direction:'outbound',weight:'distance'});

GraphFunctions

153

http://en.wikipedia.org/wiki/Betweenness_centrality


_radiusGettheradiusofagraph.

`

Parameters

options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheradiuscannotbecalculated.

Examples

Arouteplannerexample,theradiusofthegraph.



arangosh>graph._radius();

1

Arouteplannerexample,theradiusofthegraph.Thisconsiderstheactualdistances.



arangosh>graph._radius({weight:'distance'});

1

Arouteplannerexample,theradiusofthegraphregardingonlyoutboundpaths.



arangosh>graph._radius({direction:'outbound',weight:'distance'});

1

_diameter

Getthediameterofagraph.

graph._diameter(graphName,options)

Parameters

options(optional)Anobjectdefiningfurtheroptions.Canhavethefollowingvalues:direction:Thedirectionoftheedges.Possiblevaluesareoutbound,inboundandany(default).weight:Thenameoftheattributeoftheedgescontainingtheweight.defaultWeight:Onlyusedwiththeoptionweight.Ifanedgedoesnothavetheattributenamedasdefinedinoptionweightthisdefaultisusedasweight.Ifnodefaultissuppliedthedefaultwouldbepositiveinfinitysothepathandhencetheradiuscannotbecalculated.

Examples

GraphFunctions

154

http://en.wikipedia.org/wiki/Eccentricity_%28graph_theory%29

http://en.wikipedia.org/wiki/Eccentricity_%28graph_theory%29

Arouteplannerexample,thediameterofthegraph.



arangosh>graph._diameter();

1

Arouteplannerexample,thediameterofthegraph.Thisconsiderstheactualdistances.



arangosh>graph._diameter({weight:'distance'});

1

Arouteplannerexample,thediameterofthegraphregardingonlyoutboundpaths.



arangosh>graph._diameter({direction:'outbound',weight:'distance'});

1

GraphFunctions

155

SmartGraphsThisfeatureisonlyavailableintheEnterpriseEdition.

Thischapterdescribesthesmart-graphmodule.Itenablesyoutomanagegraphsatscale,itwillgiveavastperformancebenefitforallgraphsshardedinanArangoDBCluster.Onasingleserverthisfeatureispointless,henceitisonlyavailableinaclustermode.IntermsofqueryingthereisnodifferencebetweensmartandGeneralGraphs.Theformerareatransparentreplacementforthelatter.SoforqueryingthegraphpleaserefertoAQLGraphOperationsandGraphFunctionssections.TheoptimizeriscleverenoughtoidentifyifweareonaSmartGraphornot.

Thedifferenceisonlyinthemanagementsection:creatingandmodifyingtheunderlyingcollectionsofthegraph.ForadetailedAPIreferencepleaserefertoSmartGraphManagement.

Whatmakesagraphsmart?

Mostgraphshaveonefeaturethatdividestheentiregraphintoseveralsmallersubgraphs.Thesesubgraphshavealargeamountofedgesthatonlyconnectverticesinthesamesubgraphandonlyhavefewedgesconnectingverticesfromothersubgraphs.Examplesforthesegraphsare:

SocialNetworks

Typicallythefeaturehereistheregion/countryuserslivein.Everyusertypicalliyhasmorecontactsinthesameregion/countrythenshehasinotherregions/countries

TransportSystems

Forthosealsothefeatureistheregion/country.Youhavemanylocaltransportionbutonlyfewaccrosscountries.

E-Commerce

Inthiscaseprobablythecategoryofproductsisagoodfeature.Oftenproductsofthesamecategoryareboughttogether.

Ifthisfeatureisknown,SmartGraphscanmakeuseifit.WhencreatingaSmartGraphyouhavetodefineasmartAttribute,whichisthenameofanattributestoredineveryvertex.Thegraphwillthanbeautomaticallyshardedinsuchawaythatallverticeswiththesamevaluearestoredonthesamephysicalmachine,alledgesconnectingverticeswithidenticalsmartAttributevaluesarestoredonthismachineaswell.Duringquerytimethequeryoptimizerandthequeryexecutorbothknowforeverydocumentexactlywhereitisstoredandcantherebyminimizenetworkoverhead.Everythingthatcanbecomputedlocallywillbecomputedlocally.

BenefitsofSmartGraphs

Becauseoftheabovedescribedguaranteedsharding,theperformanceofqueriesthatonlycoveronesubgraphhaveaperformancealmostequaltoanonlylocalcomputation.Queriesthatcovermorethanonesubgraphrequiresomenetworkoverhead.Themoresubgraphsaretouchedthemorenetworkcostwillapply.HowevertheoverallperformanceisneverworsethanthesamequeryonaGeneralGraph.

Gettingstarted

FirstofallSmartGraphscannotuseexistingcollections,whenswitchingtoSmartGraphfromanexistingdatasetyouhavetoimportthedataintoafreshSmartGraph.Thisswitchcanbeeasilyachievedwitharangodumpandarangorestore.TheonlythingyouhavetochangeinthispipelineisthatyoucreatethenewcollectionswiththeSmartGraphbeforestartingarangorestore.

Createagraph

IncomparisontoGeneralGraphwehavetoaddmoreoptionswhencreatingthegraph.ThetwooptionssmartGraphAttributeandnumberOfShardsarerequiredandcannotbemodifedlater.

arangosh>vargraph_module=require("@arangodb/smart-graph");

arangosh>vargraph=graph_module._create("myGraph",[],[],{smartGraphAttribute:"region",numberOfShards:9});

arangosh>graph;

[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:[]]

Addsomevertexcollections

SmartGraphs

156

ThisisagainidenticaltoGeneralGraph.Themodulewillsetupcorrectshardingforallthesecollections.Note:Thecollectionshavetobenew.

arangosh>graph._addVertexCollection("shop");

arangosh>graph._addVertexCollection("customer");

arangosh>graph._addVertexCollection("pet");

arangosh>graph;

[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:["shop","customer","pet"]]

DefinerelationsontheGraph

arangosh>varrel=graph_module._relation("isCustomer",["shop"],["customer"]);

arangosh>graph._extendEdgeDefinitions(rel);

arangosh>graph;

[SmartGraphmyGraphEdgeDefinitions:["isCustomer:[shop]->[customer]"]VertexCollections:["pet"]]

SmartGraphs

157

SmartGraphManagementThischapterdescribestheJavaScriptinterfaceforcreatingandmodifyingSmartGraphs.AtfirstyouhavetonotethateverySmartGraphisaspecializedversionofaGeneralGraph,whichmeansalloftheGeneralGraphfunctionalityisavailableonaSmartGraphaswell.Themajordifferenceofbothmodulesishandlingoftheunderlyingcollections,theGeneralGraphdoesnotenforceormaintainanyshardingofthecollectionsandcanthereforcombinearbitrarysetsofexistingcollections.SmartGraphsenforceandrelyonaspecialshardingoftheunderlyingcollectionsandhencecanonlyworkwithcollectionsthatarecreatedthroughtheSmartGraphitself.ThisalsomeansthatSmartGraphscannotbeoverlapping,acollectioncaneitherbeshardedforoneSmartGraphorfortheother.IfyouneedtomakesurethatallqueriescanbeexecutedwithSmartGraphperformance,justcreateonelargeSmartGraphcoveringeverythingandqueryitstatingthesubsetofedgecollectionsexplicitly.TogenerallyunderstandtheconceptofthismodulepleasereadthechapteraboutGeneralGraphManagementfirst.Inthefollowingwewillonlydescribetheoverloadedfunctionality.Everythingelseworksidenticalinbothmodules.

Createagraph

AlsoSmartGraphsrequireedgerelationstobecreated,theformatoftherelationsisidentical.TheonlydifferenceisthatallcollectionsusedwithintherelationstocreateanewSmartGraphcannotexistyet.TheyhavetobecreatedbytheGraphinordertoenforcethecorrectsharding.

Createagraph

graph_module._create(graphName,edgeDefinitions,orphanCollections,smartOptions)

ThecreationofagraphrequiresthenameandsomeSmartGraphoptions.DuetotheAPIedgeDefinitionsandorphanCollectionshavetobegiven,butbothcanbeemptyarraysandcanbecreatedlater.TheedgeDefinitionscanbecreatedusingtheconveniencemethod_relationknownfromthegeneral-graphmodule,whichisalsoavailablehere.orphanCollectionsagainisjustalistofadditionalvertexcollectionswhicharenotyetconnectedviaedgesbutshouldfollowthesameshardingtobeconnectedlateron.Allcollectionsusedwithinthecreationprocessarenewlycreated.Theprocesswillfailifoneofthemalreadyexists.Allnewlycreatedcollectionswillimmediatelybedroppedagaininthefailedcase.

Parameters

graphName(required)UniqueidentifierofthegraphedgeDefinitions(required)Listofrelationdefinitionobjects,maybeemptyorphanCollections(required)Listofadditionalvertexcollectionnames,maybeemptysmartOptions(required)AJSONobjecthavingthefollowingkeys:

numberOfShards(required)Thenumberofshardsthatwillbecreatedforeachcollection.Tomaintainthecorrectshardingallcollectionsneedanidenticalnumberofshards.Thiscannotbemodifiedaftercreationofthegraph.smartGraphAttribute(required)Theattributethatwillbeusedforsharding.Allverticesarerequiredtohavethisattributesetandithastobeastring.Edgesderivetheattributefromtheirconnectedvertices.

Examples

Createanemptygraph,edgedefinitionscanbeaddedatruntime:


arangosh>vargraph=graph_module._create("myGraph",[],[],{smartGraphAttribute:"region",numberOfShards:9});

[SmartGraphmyGraphEdgeDefinitions:[]VertexCollections:[]]

Createagraphusinganedgecollectionedgesandasinglevertexcollectionvertices


arangosh>varedgeDefinitions=[graph_module._relation("edges","vertices","vertices")];

arangosh>vargraph=graph_module._create("myGraph",edgeDefinitions,[],{smartGraphAttribute:"region",numberOfShards:9});

[SmartGraphmyGraphEdgeDefinitions:["edges:[vertices]->[vertices]"]VertexCollections:[]]

Createagraphwithedgedefinitionsandorphancollections:


arangosh>varedgeDefinitions=[graph_module._relation("myRelation",["male","female"],["male","female"])];


158

arangosh>vargraph=graph_module._create("myGraph",edgeDefinitions,["sessions"],{smartGraphAttribute:"region",numberOfSh

ards:9});

[GraphmyGraphEdgeDefinitions:[

"myRelation:[female,male]->[female,male]"

]VertexCollections:[

"sessions"

]]

ModifyagraphdefinitionduringruntimeAfteryouhavecreatedaSmartGraphitsdefinitionisalsonotimmutable.Youcanstilladdorremoverelations.ThisisagainidenticaltoGeneralGraphs.Howeverthereisoneimportantdifference:Youcanonlyaddcollectionsthateitherdonotexist,orthathavebeencreatedbythisgraphearlier.Thelatercanbeachievedifyouforexampleremoveanorphancollectionfromthisgraph,withoutdroppingthecollectionitself.Thanaftersometimeyoudecidetoadditagain,itcanbeused.Thisisbecausetheenforcedshardingisstillappliedtothisvertexcollection,henceitissuitabletobeaddedagain.

Removeavertexcollection

Removeavertexcollectionfromthegraph

graph._removeVertexCollection(vertexCollectionName,dropCollection)

InmostcasesthisfunctionworksidenticallytotheGeneralGraphone.Butthereisonespecialcase:Thefirstvertexcollectionaddedtothegraph(eitherorphanorwithinarelation)definestheshardingforallcollectionswithinthegraph.Thiscollectioncanneverberemovedfromthegraph.

Parameters

vertexCollectionName(required)Nameofvertexcollection.dropCollection(optional)Iftruethecollectionwillbedroppedifitisnotusedinanyothergraph.Default:false.

Examples

Thefollowingexampleshowsthatyoucannotdroptheinitialcollection.Youhavetodropthecompletegraph.Ifyoujustwanttogetridofthedatatruncateit.

arangosh>vargraph_module=require("@arangodb/smart-graph")

arangosh>varrelation=graph_module._relation("edges","vertices","vertices");

arangosh>vargraph=graph_module._create("myGraph",[relation],["other"],{smartGraphAttribute:"region",numberOfShards:

9});


[

"other"

]

arangosh>graph._deleteEdgeDefinition("edges");


[

"vertices",

"other"

]

arangosh>graph._removeVertexCollection("other");


[

"vertices"

]

arangosh>graph._removeVertexCollection("vertices");

ArangoError4002:cannotdropthissmartcollection


159

TraversalsArangoDBprovidesseveralwaystoquerygraphdata.Verysimpleoperationscanbecomposedwiththelow-leveledgemethodsedges,inEdges,andoutEdgesforedgecollections.Theseworkonnamedandanonymousgraphs.Formorecomplexoperations,ArangoDBprovidespredefinedtraversalobjects.

AlsoTraversalshavebeenaddedtoAQL.PleasereadthechapteraboutAQLtraversersalsbeforeyoucontinuereadinghere.MostofthetraversalcasesarecoveredbyAQLandwillbeexecutedinanoptimizedway.OnlyifthelogicforyouristoocomplextobedefinedusingAQLfiltersyoucanusethetraversalobjectdefinedherewhichgivesyoucompleteprogrammaticaccesstothedata.

Foranyofthefollowingexamples,we'llbeusingtheexamplecollectionsvande,populatedwithcontinents,countriesandcapitalsdatalistedbelow(seeExampleData).

StartingfromScratchArangoDBprovidestheedges,inEdges,andoutEdgesmethodsforedgecollections.Thesemethodscanbeusedtoquicklydetermineifavertexisconnectedtoothervertices,andwhich.ThisfunctionalitycanbeexploitedtowriteverysimplegraphqueriesinJavaScript.

Forexample,todeterminewhichedgesarelinkedtotheworldvertex,wecanuseinEdges:

db.e.inEdges('v/world').forEach(function(edge){

require("@arangodb").print(edge._from,"->",edge.type,"->",edge._to);

});

inEdgeswillgiveusallingoingedgesforthespecifiedvertexv/world.TheresultisaJavaScriptarraythatwecaniterateoverandprinttheresults:

v/continent-africa->is-in->v/world

v/continent-south-america->is-in->v/world

v/continent-asia->is-in->v/world

v/continent-australia->is-in->v/world

v/continent-europe->is-in->v/world

v/continent-north-america->is-in->v/world

Note:edges,inEdges,andoutEdgesreturnanarrayofedges.Ifwewanttoretrievethelinkedvertices,wecanuseeachedges'_fromand_toattributesasfollows:

db.e.inEdges('v/world').forEach(function(edge){

require("@arangodb").print(db._document(edge._from).name,"->",edge.type,"->",db._document(edge._to).name);

});

Weareusingthedocumentmethodfromthedbobjecttoretrievetheconnectedverticesnow.

Whilethismaybesufficientforone-levelgraphoperations,writingatraversalbyhandmaybecometoocomplexformulti-leveltraversals.

Traversals

160

GettingstartedTouseatraversalobject,wefirstneedtorequirethetraversalmodule:

vartraversal=require("@arangodb/graph/traversal");

varexamples=require("@arangodb/graph-examples/example-graph.js");

examples.loadGraph("worldCountry");

Wethenneedtosetupaconfigurationforthetraversalanddetermineatwhichvertextostartthetraversal:

varconfig={

datasource:traversal.generalGraphDatasourceFactory("worldCountry"),

strategy:"depthfirst",

order:"preorder",

filter:traversal.visitAllFilter,

expander:traversal.inboundExpander,

maxDepth:1

};

varstartVertex=db._document("v/world");

Note:ThestartVertexneedstobeadocument,notonlyadocumentid.

Wecanthencreateatraverserandstartthetraversalbycallingitstraversemethod.Notethattraverseneedsaresultobject,whichitcanmodifyinplace:

varresult={

visited:{

vertices:[],

paths:[]

}

};

vartraverser=newtraversal.Traverser(config);

traverser.traverse(result,startVertex);

Finally,wecanprintthecontentsoftheresultsobject,limitedtothevisitedvertices.Wewillonlyprintthenameandtypeofeachvisitedvertexforbrevity:

require("@arangodb").print(result.visited.vertices.map(function(vertex){

returnvertex.name+"("+vertex.type+")";

}));

Thefullscript,whichincludesallstepscarriedoutsofaristhus:


varconfig={

datasource:traversal.generalGraphDatasourceFactory("worldCountry"),


order:"preorder",



maxDepth:1

};

varstartVertex=db._document("v/world");

varresult={

visited:{

vertices:[],

paths:[]

}

};




161



}));

Theresultisanarrayofverticesthatwerevisitedduringthetraversal,startingatthestartvertex(i.e.v/worldinourexample):

[

"World(root)",

"Africa(continent)",

"Asia(continent)",

"Australia(continent)",

"Europe(continent)",

"NorthAmerica(continent)",

"SouthAmerica(continent)"

]

Note:Theresultislimitedtoverticesdirectlyconnectedtothestartvertex.WeachievedthisbysettingthemaxDepthattributeto1.Notsettingitwouldreturnthefullarrayofvertices.

TraversalDirection

Fortheexamplescontainedinthismanual,we'llbestartingthetraversalsatvertexv/world.Verticesinourgraphareconnectedlikethis:

v/world<-is-in<-continent(Africa)<-is-in<-country(Algeria)<-is-in<-capital(Algiers)

Togetanymeaningfulresults,wemusttraversethegraphininboundorder.Thismeans,we'llbefollowingallincomingedgesoftoavertex.Inthetraversalconfiguration,wehavespecifiedthisviatheexpanderattribute:

varconfig={

...

expander:traversal.inboundExpander

};

Forothergraphs,wemightwanttotraverseviatheoutgoingedges.Forthis,wecanusetheoutboundExpander.ThereisalsoananyExpander,whichwillfollowbothoutgoingandincomingedges.Thisshouldbeusedwithcareandthetraversalshouldalwaysbelimitedtoamaximumnumberofiterations(e.g.usingthemaxIterationsattribute)inordertoterminateatsomepoint.

Toinvokethedefaultoutboundexpanderforagraph,simplyusethepredefinedfunction:

varconfig={

...

expander:traversal.outboundExpander

};

Pleasenotetheoutboundexpanderwillnotproduceanyoutputfortheexamplesifwestillstartthetraversalatthev/worldvertex.

Still,wecanusetheoutboundexpanderifwestartsomewhereelseinthegraph,e.g.


varconfig={

datasource:traversal.generalGraphDatasourceFactory("world_graph"),


order:"preorder",


expander:traversal.outboundExpander

};

varstartVertex=db._document("v/capital-algiers");

varresult={

visited:{

vertices:[],

paths:[]

}

};


162





}));

Theresultis:

[

"Algiers(capital)",

"Algeria(country)",


"World(root)"

]

whichconfirmsthatnowwe'regoingoutbound.

TraversalStrategy

Depth-firsttraversals

Thevisitationorderofverticesisdeterminedbythestrategyandorderattributessetintheconfiguration.Wechosedepthfirstandpreorder,meaningthetraverserwillvisiteachvertexbeforehandlingconnectededges(pre-order),anddescendintoanyconnectededgesbeforeprocessingotherverticesonthesamelevel(depth-first).

Let'sremovethemaxDepthattributenow.We'llnowbegettingallvertices(directlyandindirectlyconnectedtothestartvertex):

varconfig={



order:"preorder",


expander:traversal.inboundExpander

};

varresult={

visited:{

vertices:[],

paths:[]

}

};





}));

Theresultwillbealongerarray,assembledindepth-first,pre-orderorder.Foreachcontinentfound,thetraverserwilldescendintolinkedcountries,andthenintothelinkedcapital:

[

"World(root)",


"Algeria(country)",

"Algiers(capital)",

"Angola(country)",

"Luanda(capital)",

"Botswana(country)",

"Gaborone(capital)",

"BurkinaFaso(country)",

"Ouagadougou(capital)",

...

]


163

Let'sswitchtheorderattributefrompreordertopostorder.Thiswillmakethetraverservisitverticesafterallconnectedverticeswerevisited(i.e.mostdistantverticeswillbeemittedfirst):

[

"Algiers(capital)",

"Algeria(country)",

"Luanda(capital)",

"Angola(country)",

"Gaborone(capital)",

"Botswana(country)",

"Ouagadougou(capital)",


"Bujumbura(capital)",

"Burundi(country)",

"Yaounde(capital)",

"Cameroon(country)",

"N'Djamena(capital)",

"Chad(country)",

"Yamoussoukro(capital)",

"Coted'Ivoire(country)",

"Cairo(capital)",

"Egypt(country)",

"Asmara(capital)",

"Eritrea(country)",


...

]

Breadth-firsttraversals

Ifwegobacktopreorder,butchangethestrategytobreadth-firstandre-runthetraversal,we'llseethatthereturnorderchanges,anditemsonthesamelevelwillbereturnedadjacently:

[

"World(root)",


"Asia(continent)",




"SouthAmerica(continent)",


"Burundi(country)",

"Cameroon(country)",

"Chad(country)",

"Algeria(country)",

"Angola(country)",

...

]

Note:Theorderofitemsreturnedforthesamelevelisundefined.Thisisbecausethereisnonaturalorderofedgesforavertexwithmultipleconnectededges.Toexplicitlysettheorderforedgesonthesamelevel,youcanspecifyanedgecomparatorfunctionwiththesortattribute:

varconfig={

...

sort:function(l,r){returnl._key<r._key?1:-1;}

...

};

Theargumentslandrareedgedocuments.Thiswilltraverseedgesofthesamevertexinbackward_keyorder:

[

"World(root)",

"SouthAmerica(continent)",




"Asia(continent)",


164


"Ecuador(country)",

"Colombia(country)",

"Chile(country)",

"Brazil(country)",

"Bolivia(country)",

"Argentina(country)",

...

]

Note:Thisattributeonlyworksfortheusualexpanderstraversal.inboundExpander,traversal.outboundExpander,traversal.anyExpanderandtheircorresponding"WithLabels"variants.Ifyouareusingcustomexpandersyouhavetoorganizethesortingwithinthespecifiedexpander.

WritingCustomVisitors

Sofarwehaveusedmuchofthetraverser'sdefaultfunctions.Thetraverserisveryconfigurableandmanyofthedefaultfunctionscanbeoverriddenwithcustomfunctionality.

Forexample,wehavebeenusingthedefaultvisitorfunction(whichisalwaysusediftheconfigurationdoesnotcontainthevisitorattribute).Thedefaultvisitorfunctioniscalledforeachvertexinatraversal,andwillpushitintotheresult.Thisisthereasonwhytheresultvariablelookeddifferentafterthetraversal,andneededtobeinitializedbeforethetraversalwasstarted.

Notethatthedefaultvisitor(namedtrackingVisitor)willaddeveryvisitedvertexintotheresult,includingthefullpathsfromthestartvertex.Thisisusefulforlearninganddebuggingpurposes,butshouldbeavoidedinproductionbecauseitmightproduce(andcopy)hugeamountsofdata.Instead,onlythosedatashouldbecopiedintotheresultthatareactuallynecessary.

Thetraversercomeswiththefollowingpredefinedvisitors:

trackingVisitor:thisisthedefaultvisitor.Itwillcopyalldataofeachvisitedvertexplusthefullpathinformationintotheresult.Thiscanbeslowiftheresultsetishugeorverticescontainalotofdata.countingVisitor:thisisaverylightweightvisitor:allitdoesisincreaseacounterintheresultforeachvertexvisited.Vertexdataandpathswillnotbecopiedintotheresult.doNothingVisitor:ifnoactionshallbecarriedoutwhenavertexisvisited,thisvisitorcanbeemployed.Itwillnotdoanythingandwillthusbefast.Itcanbeusedforperformancecomparisonswithothervisitors.

Wecanalsowriteourownvisitorfunctionifwewantto.Thegeneralfunctionsignatureforvisitorfunctionsisasfollows:

varconfig={

...

visitor:function(config,result,vertex,path,connected){...}

};

Note:theconnectedparametervaluewillonlybesetifthetraversalorderissettopreorder-expander.Otherwise,thisparameterwon'tbesetbythetraverser.

Visitorfunctionsarenotexpectedtoreturnanyvalues.Instead,theycanmodifytheresultvariable(e.g.bypushingthecurrentvertexintoit),ordoanythingelse.Forexample,wecancreateasimplevisitorfunctionthatonlyprintsinformationaboutthecurrentvertexaswetraverse:

varconfig={



order:"preorder",



visitor:function(config,result,vertex,path){

require("@arangodb").print("visitingvertex",vertex.name);

}

};


traverser.traverse(undefined,startVertex);

Towriteavisitorthatincrementsacountereachtimeavertexisvisited,wecouldwritethefollowingcustomvisitor:


165

config.visitor=function(config,result,vertex,path,connected){

if(!result){

result={};

}

if(!result.hasOwnProperty('count')){

result.count=0;

}

++result.count;

}

Notethatsuchvisitorisalreadypredefined(it'sthecountingVisitordescribedabove).Itcanbeusedasfollows:

config.visitor=traversal.countingVisitor;

Anotherexampleofavisitorisonethatcollectsthe_idvaluesofallverticesvisited:


if(!result){

result={};

}

if(!result.hasOwnProperty("visited")){

result.visited={vertices:[]};

}

result.visited.vertices.push(vertex._id);

}

Whenthetraversalorderissettopreorder-expander,thetraverserwillpassafifthparametervalueintothevisitorfunction.Thisparametercontainstheconnectededgesofthevisitedvertexasanarray.Thiscanbehandybecauseinthiscasethevisitorwillgetallinformationaboutthevertexandtheconnectededgestogether.

Forexample,thefollowingvisitorcanbeusedtoprintonlyleafnodes(thatdonothaveanyfurtherconnectededges):


if(connected&&connected.length===0){

require("@arangodb").print("foundaleaf-node:",vertex);

}

}

Notethatforthisvisitortowork,thetraversalorderattributeneedstobesettothevaluepreorder-expander.

FilteringVerticesandEdges

FilteringVertices

Sofarwehaveprintedorreturnedallverticesthatwerevisitedduringthetraversal.Thisisnotalwaysrequired.Iftheresultshallberestricttojustspecificvertices,wecanuseafilterfunctionforvertices.Itcanbedefinedbysettingthefilterattributeofatraversalconfiguration,e.g.:

varconfig={

filter:function(config,vertex,path){

if(vertex.type!=='capital'){

return'exclude';

}

}

}

Theabovefilterfunctionwillexcludeallverticesthatdonothaveatypevalueofcapital.Thefilterfunctionwillbecalledforeachvertexfoundduringthetraversal.Itwillreceivethetraversalconfiguration,thecurrentvertex,andthefullpathfromthetraversalstartvertextothecurrentvertex.Thepathconsistsofanarrayofedges,andanarrayofvertices.Wecouldalsofiltereverythingbutcapitalsby


166

checkingthelengthofthepathfromthestartvertextothecurrentvertex.Capitalswillhaveadistanceof3fromthev/worldstartvertex(capital→is-in→country→is-in→continent→is-in→world):

varconfig={

...


if(path.edges.length<3){

return'exclude';

}

}

}

Note:Ifafilterfunctionreturnsnothing(orundefined),thecurrentvertexwillbeincluded,andallconnectededgeswillbefollowed.Ifafilterfunctionreturnsexcludethecurrentvertexwillbeexcludedfromtheresult,andallstillallconnectededgeswillbefollowed.Ifafilterfunctionreturnsprune,thecurrentvertexwillbeincluded,butnoconnectededgeswillbefollowed.

Forexample,thefollowingfilterfunctionwillnotdescendintoconnectededgesofcontinents,limitingthedepthofthetraversal.Still,continentverticeswillbeincludedintheresult:

varconfig={

...


if(vertex.type==='continent'){

return'prune';

}

}

}

Itisalsopossibletocombineexcludeandprunebyreturninganarraywithbothvalues:

return['exclude','prune'];

FilteringEdges

Itispossibletoexcludecertainedgesfromthetraversal.Tofilteronedges,afilterfunctioncanbedefinedviatheexpandFilterattribute.TheexpandFilterisafunctionwhichiscalledforeachedgeduringatraversal.

Itwillreceivethecurrentedge(edgevariable)andthevertexwhichtheedgeconnectsto(inthedirectionofthetraversal).Italsoreceivesthecurrentpathfromthestartvertexuptothecurrentvertex(excludingthecurrentedgeandthevertextheedgepointsto).

Ifthefunctionreturnstrue,theedgewillbefollowed.Ifthefunctionreturnsfalse,theedgewillnotbefollowed.Hereisaverysimplecustomedgefilterfunctionimplementation,whichsimplyincludesedgesifthe(edges)pathlengthislessthan1,andwillexcludeanyotheredges.Thiswilleffectivelyterminatethetraversalafterthefirstlevelofedges:

varconfig={

...

expandFilter:function(config,vertex,edge,path){

return(path.edges.length<1);

}

};

WritingCustomExpanders

Theedgesconnectedtoavertexaredeterminedbytheexpander.Sofarwehaveusedadefaultexpander(thedefaultinboundexpandertobeprecise).Thedefaultinboundexpandersimplyenumeratesallconnectedingoingedgesforavertex,basedontheedgecollectionspecifiedinthetraversalconfiguration.

Thereisalsoadefaultoutboundexpander,whichwillenumerateallconnectedoutgoingedges.Finally,thereisananyexpander,whichwillfollowbothingoingandoutgoingedges.

Ifconnectededgesmustbedeterminedinsomedifferentfashionforwhateverreason,acustomexpandercanbewrittenandregisteredbysettingtheexpanderattributeoftheconfiguration.Theexpanderfunctionsignatureisasfollows:


167

varconfig={

...

expander:function(config,vertex,path){...}

}

Itistheexpander'sresponsibilitytoreturnalledgesandverticesdirectlyconnectedtothecurrentvertex(whichispassedviathevertexvariable).Thefullpathfromthestartvertexuptothecurrentvertexisalsosuppliedviathepathvariable.Anexpanderisexpectedtoreturnanarrayofobjects,whichneedtohaveanedgeandavertexattributeeach.

Note:Ifyouwanttorelyonaparticularorderinwhichtheedgesaretraversed,youhavetosorttheedgesreturnedbyyourexpanderwithinthecodeoftheexpander.Thefunctionstogetoutbound,inboundoranyedgesfromavertexdonotguaranteeanyparticularorder!

Acustomimplementationofaninboundexpandercouldlooklikethis(thisisanon-deterministicexpander,whichrandomlydecideswhetherornottoincludeconnectededges):

varconfig={

...

expander:function(config,vertex,path){

varconnected=[];

vardatasource=config.datasource;

datasource.getInEdges(vertex._id).forEach(function(edge){

if(Math.random()>=0.5){

connected.push({edge:edge,vertex:(edge._from)});

}

});

returnconnected;

}

};

Acustomexpandercanalsobeusedasanedgefilterbecauseithasfullcontroloverwhichedgeswillbereturned.

Followingaretwoexamplesofcustomexpandersthatpickedgesbasedonattributesoftheedgesandtheconnectedvertices.

Findingtheconnectededges/verticesbasedonanattributewhenintheconnectedvertices.Thegoalistofollowtheedgethatleadstothevertexwiththehighestvalueinthewhenattribute:

varconfig={

...



//determinealloutgoingedges

varoutEdges=datasource.getOutEdges(vertex);

if(outEdges.length===0){

return[];

}

vardata=[];

outEdges.forEach(function(edge){

data.push({edge:edge,vertex:datasource.getInVertex(edge)});

});

//sortoutgoingverticesaccordingto"when"attributevalue

data.sort(function(l,r){

if(l.vertex.when===r.vertex.when){

return0;

}

return(l.vertex.when<r.vertex.when?1:-1);

});

//pickfirstvertexfound(withhighest"when"attributevalue)

return[data[0]];

}

...

};

Findingtheconnectededges/verticesbasedonanattributewhenintheedgeitself.Thegoalistopicktheoneedge(outofpotentiallymany)thathasthehighestwhenattributevalue:


168

varconfig={

...



//determinealloutgoingedges

varoutEdges=datasource.getOutEdges(vertex);

if(outEdges.length===0){

return[];//returnanemptyarray

}

//sortalloutgoingedgesaccordingto"when"attribute

outEdges.sort(function(l,r){

if(l.when===r.when){

return0;

}

return(l.when<r.when?-1:1);

});

//returnfirstedge(theonewithhighest"when"value)

varedge=outEdges[0];

try{

varv=datasource.getInVertex(edge);

return[{edge:edge,vertex:v}];

}

catch(e){}

return[];

}

...

};

HandlingUniqueness

Graphsmaycontaincycles.Tobeontopofwhathappenswhenatraversalencountersavertexoranedgeithasalreadyvisited,thereareconfigurationoptions.

Thedefaultconfigurationistovisiteveryvertex,regardlessofwhetheritwasalreadyvisitedinthesametraversal.However,edgeswillbydefaultonlybefollowediftheyarenotalreadypresentinthecurrentpath.

Imaginethefollowinggraphwhichcontainsacycle:

A->B->C->A

WhenthetraversalfindstheedgefromCtoA,itwillbydefaultfollowit.Thisisbecausewehavenotseenthisedgeyet.ItwillalsovisitvertexAagain.Thisisbecausebydefaultallverticeswillbevisited,regardlessofwhetheralreadyvisitedornot.

However,thetraversalwillnotagainfollowingtheoutgoingedgefromAtoB.ThisisbecausewealreadyhavetheedgefromAtoBinourcurrentpath.

Thesedefaultsettingswillpreventinfinitetraversals.

Toadjusttheuniquenessforvisitingvertices,therearethefollowingoptionsforuniqueness.vertices:

"none":alwaysvisitavertices,regardlessofwhetheritwasalreadyvisitedornot"global":visitavertexonlyifitwasnotvisitedinthetraversal"path":visitavertexifitisnotincludedinthecurrentpath

Toadjusttheuniquenessforfollowingedges,therearethefollowingoptionsforuniqueness.edges:

"none":alwaysfollowanedge,regardlessofwhetheritwasfollowedbefore"global":followanedgeonlyifitwasn'tfollowedinthetraversal"path":followanedgeifitisnotincludedinthecurrentpath

Notethatuniquenesscheckingwillhavesomeeffectonbothruntimeandmemoryusage.Forexample,whenuniquenesschecksaresetto"global" ,arraysofvisitedverticesandedgesmustbekeptinmemorywhilethetraversalisexecuted.Globaluniquenessshouldthusonlybeusedwhenatraversalisexpectedtovisitfewnodes.


169

Intermsofruntime,turningoffuniquenesschecks(bysettingbothoptionsto"none")isthebestchoice,butitisonlysafeforgraphsthatdonotcontaincycles.Whenuniquenesschecksaredeactivatedinagraphwithcycles,thetraversalmightnotabortinasensibleamountoftime.

Optimizations

Thereareafewoptionsformakingatraversalrunfaster.

Thebestoptionistomaketheamountofvisitedverticesandfollowededgesassmallaspossible.Thiscanbeachievedbywritingcustomfilterandexpanderfunctions.Suchfunctionsshouldonlyincludeverticesofinterest,andonlyfollowedgesthatmightbeinteresting.

TraversaldepthcanalsobeboundedwiththeminDepthandmaxDepthoptions.

Anotherwaytospeeduptraversalsistowriteacustomvisitorfunction.Thedefaultvisitorfunction(trackingVisitor)willcopyeveryvisitedvertexintotheresult.Ifverticescontainlotsofdata,thismightbeexpensive.Itisthereforerecommendedtoonlycopysuchdataintotheresultthatisactuallyneeded.Thedefaultvisitorfunctionwillalsocopythefullpathtothevisiteddocumentintotheresult.Thisisevenmoreexpensiveandshouldbeavoidedifpossible.

Ifthegoalofatraversalistoonlycountthenumberofvisitedvertices,theprefabcountingVisitorwillbemuchmoreefficientthanthedefaultvisitor.

Forgraphsthatareknowntonotcontainanycycles,uniquenesschecksshouldbeturnedoff.Thiscanachievedviatheuniquenessconfigurationoptions.Notethatuniquenesschecksshouldnotbeturnedoffforgraphsthatareknowncontaincyclesorifthereisnoinformationaboutthegraph'sstructure.

Bydefault,atraversalwillonlyprocessalimitednumberofvertices.Thisisprotecttheuserfromunintentionallyrunanever-endingtraversalonagraphwithcyclicdata.HowmanyverticeswillbeprocessedatmostisdeterminedbythemaxIterationsconfigurationoption.IfatraversalhitsthecapspecifiedbymaxIterations,itwillabortandthrowatoomanyiterationsexception.Ifthiserrorisencountered,themaxIterationsvalueshouldbeincreasedifitismadesurethattheothertraversalconfigurationparametersaresaneandthetraversalwillabortnaturallyatsomepoint.

Finally,thebuildVerticesconfigurationoptioncanbesettofalsetoavoidlookingupandfullyconstructingvertexdata.Ifallthat'sneededfromverticesarethe_idor_keyattributes,thebuildverticesoptioncanbesettofalse.Ifvisitor,filterorexpandFilterfunctionsneedtoaccessothervertexattributes,theoptionshouldnotbechanged.

ConfigurationOverview

Thissectionsummarizestheconfigurationattributesforthetraversalobject.Theconfigurationcanconsistofthefollowingattributes:

visitor:visitorfunctionforvertices.Itwillbecalledforallnon-excludedvertices.Thegeneralvisitorfunctionsignatureisfunction(config,result,vertex,path).Ifthetraversalorderispreorder-expander,theconnectingedgesofthevisitedvertexwillbepassedasthefifthparameter,extendingthefunctionsignatureto:function(config,result,vertex,path,edges).

Visitorfunctionsarenotexpectedtoreturnvalues,buttheymaymodifytheresultvariableasneeded(e.g.bypushingvertexdataintotheresult).

expander:expanderfunctionthatisresponsibleforreturningedgesandverticesdirectlyconnectedtoavertex.Thefunctionsignatureisfunction(config,vertex,path).Theexpanderfunctionisrequiredtoreturnanarrayofconnectionobjects,consistingofanedgeandvertexattributeeach.Iftherearenoconnectingedges,theexpanderisexpectedtoreturnanemptyarray.filter:vertexfilterfunction.Thefunctionsignatureisfunction(config,vertex,path).Itmayreturnoneofthefollowingvalues:

undefined:vertexwillbeincludedintheresultandconnectededgeswillbetraversed"exclude":vertexwillnotbeincludedintheresultandconnectededgeswillbetraversed"prune":vertexwillbeincludedintheresultbutconnectededgeswillnotbetraversed["prune" ,"exclude" ]:vertexwillnotbeincludedintheresultandconnectededgeswillnotbereturned

expandFilter:filterfunctionappliedoneachedge/vertexcombinationdeterminedbytheexpander.Thefunctionsignatureisfunction(config,vertex,edge,path).Thefunctionshouldreturntrueiftheedge/vertexcombinationshouldbeprocessed,andfalseifitshouldbeignored.sort:afilterfunctiontodeterminetheorderinwhichconnectededgesareprocessed.Thefunctionsignatureisfunction(l,r).Thefunctionisrequiredtoreturnoneofthefollowingvalues:

-1iflshouldhaveasortvaluelessthanr


170

1iflshouldhaveahighersortvaluethanr0iflandrhavethesamesortvalue

strategy:determinesthevisitationstrategy.Possiblevaluesaredepthfirstandbreadthfirst.order:determinesthevisitationorder.Possiblevaluesarepreorder,postorder,andpreorder-expander.preorder-expanderisthesameaspreorder,exceptthatthesignatureofthevisitorfunctionwillchangeasdescribedabove.itemOrder:determinestheorderinwhichconnectionsreturnedbytheexpanderwillbeprocessed.Possiblevaluesareforwardandbackward.maxDepth:ifsettoavaluegreaterthan0,thiswilllimitthetraversaltothismaximumdepth.minDepth:ifsettoavaluegreaterthan0,allverticesfoundonalevelbelowtheminDepthlevelwillnotbeincludedintheresult.maxIterations:themaximumnumberofiterationsthatthetraversalisallowedtoperform.Itissensibletosetthisnumbersounboundedtraversalswillterminateatsomepoint.uniqueness:anobjectthatdefineshowrepeatedvisitationsofverticesshouldbehandled.Theuniquenessobjectcanhaveasub-attributevertices,andasub-attributeedges.Eachsub-attributecanhaveoneofthefollowingvalues:

"none":nouniquenessconstraints"path":elementisexcludedifitisalreadycontainedinthecurrentpath.Thissettingmaybesensibleforgraphsthatcontaincycles(e.g.A→B→C→A)."global":elementisexcludedifitwasalreadyfound/visitedatanypointduringthetraversal.

buildVertices:thisattributecontrolswhetherverticesencounteredduringthetraversalwillbelookedupinthedatabaseandwillbemadeavailabletovisitor,filter,andexpandFilterfunctions.Bydefault,verticeswillbelookedupandmadeavailable.However,therearesomespecialusecaseswhenfullyconstructingvertexobjectsisnotnecessaryandcanbeavoided.Forexample,ifatraversalismeanttoonlycountthenumberofvisitedverticesbutdonotreadanydatafromvertices,thisoptionmightbesettotrue.


171

ExampleDataThefollowingexamplesalluseavertexcollectionvandanedgecollectione.Thevertexcollectionvcontainscontinents,countries,andcapitals.Theedgecollectionecontainsconnectionsbetweencontinentsandcountries,andbetweencountriesandcapitals.

Tosetupthecollectionsandpopulatethemwithinitialdata,thefollowingscriptwasused:

db._create("v");

db._createEdgeCollection("e");

//vertices:rootnode

db.v.save({_key:"world",name:"World",type:"root"});

//vertices:continents

db.v.save({_key:"continent-africa",name:"Africa",type:"continent"});

db.v.save({_key:"continent-asia",name:"Asia",type:"continent"});

db.v.save({_key:"continent-australia",name:"Australia",type:"continent"});

db.v.save({_key:"continent-europe",name:"Europe",type:"continent"});

db.v.save({_key:"continent-north-america",name:"NorthAmerica",type:"continent"});

db.v.save({_key:"continent-south-america",name:"SouthAmerica",type:"continent"});

//vertices:countries

db.v.save({_key:"country-afghanistan",name:"Afghanistan",type:"country",code:"AFG"});

db.v.save({_key:"country-albania",name:"Albania",type:"country",code:"ALB"});

db.v.save({_key:"country-algeria",name:"Algeria",type:"country",code:"DZA"});

db.v.save({_key:"country-andorra",name:"Andorra",type:"country",code:"AND"});

db.v.save({_key:"country-angola",name:"Angola",type:"country",code:"AGO"});

db.v.save({_key:"country-antigua-and-barbuda",name:"AntiguaandBarbuda",type:"country",code:"ATG"});

db.v.save({_key:"country-argentina",name:"Argentina",type:"country",code:"ARG"});

db.v.save({_key:"country-australia",name:"Australia",type:"country",code:"AUS"});

db.v.save({_key:"country-austria",name:"Austria",type:"country",code:"AUT"});

db.v.save({_key:"country-bahamas",name:"Bahamas",type:"country",code:"BHS"});

db.v.save({_key:"country-bahrain",name:"Bahrain",type:"country",code:"BHR"});

db.v.save({_key:"country-bangladesh",name:"Bangladesh",type:"country",code:"BGD"});

db.v.save({_key:"country-barbados",name:"Barbados",type:"country",code:"BRB"});

db.v.save({_key:"country-belgium",name:"Belgium",type:"country",code:"BEL"});

db.v.save({_key:"country-bhutan",name:"Bhutan",type:"country",code:"BTN"});

db.v.save({_key:"country-bolivia",name:"Bolivia",type:"country",code:"BOL"});

db.v.save({_key:"country-bosnia-and-herzegovina",name:"BosniaandHerzegovina",type:"country",code:"BIH"});

db.v.save({_key:"country-botswana",name:"Botswana",type:"country",code:"BWA"});

db.v.save({_key:"country-brazil",name:"Brazil",type:"country",code:"BRA"});

db.v.save({_key:"country-brunei",name:"Brunei",type:"country",code:"BRN"});

db.v.save({_key:"country-bulgaria",name:"Bulgaria",type:"country",code:"BGR"});

db.v.save({_key:"country-burkina-faso",name:"BurkinaFaso",type:"country",code:"BFA"});

db.v.save({_key:"country-burundi",name:"Burundi",type:"country",code:"BDI"});

db.v.save({_key:"country-cambodia",name:"Cambodia",type:"country",code:"KHM"});

db.v.save({_key:"country-cameroon",name:"Cameroon",type:"country",code:"CMR"});

db.v.save({_key:"country-canada",name:"Canada",type:"country",code:"CAN"});

db.v.save({_key:"country-chad",name:"Chad",type:"country",code:"TCD"});

db.v.save({_key:"country-chile",name:"Chile",type:"country",code:"CHL"});

db.v.save({_key:"country-colombia",name:"Colombia",type:"country",code:"COL"});

db.v.save({_key:"country-cote-d-ivoire",name:"Coted'Ivoire",type:"country",code:"CIV"});

db.v.save({_key:"country-croatia",name:"Croatia",type:"country",code:"HRV"});

db.v.save({_key:"country-czech-republic",name:"CzechRepublic",type:"country",code:"CZE"});

db.v.save({_key:"country-denmark",name:"Denmark",type:"country",code:"DNK"});

db.v.save({_key:"country-ecuador",name:"Ecuador",type:"country",code:"ECU"});

db.v.save({_key:"country-egypt",name:"Egypt",type:"country",code:"EGY"});

db.v.save({_key:"country-eritrea",name:"Eritrea",type:"country",code:"ERI"});

db.v.save({_key:"country-finland",name:"Finland",type:"country",code:"FIN"});

db.v.save({_key:"country-france",name:"France",type:"country",code:"FRA"});

db.v.save({_key:"country-germany",name:"Germany",type:"country",code:"DEU"});

db.v.save({_key:"country-people-s-republic-of-china",name:"People'sRepublicofChina",type:"country",code:"CHN"});

//vertices:capitals

db.v.save({_key:"capital-algiers",name:"Algiers",type:"capital"});

db.v.save({_key:"capital-andorra-la-vella",name:"AndorralaVella",type:"capital"});

db.v.save({_key:"capital-asmara",name:"Asmara",type:"capital"});

db.v.save({_key:"capital-bandar-seri-begawan",name:"BandarSeriBegawan",type:"capital"});

db.v.save({_key:"capital-beijing",name:"Beijing",type:"capital"});

db.v.save({_key:"capital-berlin",name:"Berlin",type:"capital"});

db.v.save({_key:"capital-bogota",name:"Bogota",type:"capital"});

db.v.save({_key:"capital-brasilia",name:"Brasilia",type:"capital"});

ExampleData

172

db.v.save({_key:"capital-bridgetown",name:"Bridgetown",type:"capital"});

db.v.save({_key:"capital-brussels",name:"Brussels",type:"capital"});

db.v.save({_key:"capital-buenos-aires",name:"BuenosAires",type:"capital"});

db.v.save({_key:"capital-bujumbura",name:"Bujumbura",type:"capital"});

db.v.save({_key:"capital-cairo",name:"Cairo",type:"capital"});

db.v.save({_key:"capital-canberra",name:"Canberra",type:"capital"});

db.v.save({_key:"capital-copenhagen",name:"Copenhagen",type:"capital"});

db.v.save({_key:"capital-dhaka",name:"Dhaka",type:"capital"});

db.v.save({_key:"capital-gaborone",name:"Gaborone",type:"capital"});

db.v.save({_key:"capital-helsinki",name:"Helsinki",type:"capital"});

db.v.save({_key:"capital-kabul",name:"Kabul",type:"capital"});

db.v.save({_key:"capital-la-paz",name:"LaPaz",type:"capital"});

db.v.save({_key:"capital-luanda",name:"Luanda",type:"capital"});

db.v.save({_key:"capital-manama",name:"Manama",type:"capital"});

db.v.save({_key:"capital-nassau",name:"Nassau",type:"capital"});

db.v.save({_key:"capital-n-djamena",name:"N'Djamena",type:"capital"});

db.v.save({_key:"capital-ottawa",name:"Ottawa",type:"capital"});

db.v.save({_key:"capital-ouagadougou",name:"Ouagadougou",type:"capital"});

db.v.save({_key:"capital-paris",name:"Paris",type:"capital"});

db.v.save({_key:"capital-phnom-penh",name:"PhnomPenh",type:"capital"});

db.v.save({_key:"capital-prague",name:"Prague",type:"capital"});

db.v.save({_key:"capital-quito",name:"Quito",type:"capital"});

db.v.save({_key:"capital-saint-john-s",name:"SaintJohn's",type:"capital"});

db.v.save({_key:"capital-santiago",name:"Santiago",type:"capital"});

db.v.save({_key:"capital-sarajevo",name:"Sarajevo",type:"capital"});

db.v.save({_key:"capital-sofia",name:"Sofia",type:"capital"});

db.v.save({_key:"capital-thimphu",name:"Thimphu",type:"capital"});

db.v.save({_key:"capital-tirana",name:"Tirana",type:"capital"});

db.v.save({_key:"capital-vienna",name:"Vienna",type:"capital"});

db.v.save({_key:"capital-yamoussoukro",name:"Yamoussoukro",type:"capital"});

db.v.save({_key:"capital-yaounde",name:"Yaounde",type:"capital"});

db.v.save({_key:"capital-zagreb",name:"Zagreb",type:"capital"});

//edges:continent->world

db.e.save("v/continent-africa","v/world",{type:"is-in"});

db.e.save("v/continent-asia","v/world",{type:"is-in"});

db.e.save("v/continent-australia","v/world",{type:"is-in"});

db.e.save("v/continent-europe","v/world",{type:"is-in"});

db.e.save("v/continent-north-america","v/world",{type:"is-in"});

db.e.save("v/continent-south-america","v/world",{type:"is-in"});

//edges:country->continent

db.e.save("v/country-afghanistan","v/continent-asia",{type:"is-in"});

db.e.save("v/country-albania","v/continent-europe",{type:"is-in"});

db.e.save("v/country-algeria","v/continent-africa",{type:"is-in"});

db.e.save("v/country-andorra","v/continent-europe",{type:"is-in"});

db.e.save("v/country-angola","v/continent-africa",{type:"is-in"});

db.e.save("v/country-antigua-and-barbuda","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-argentina","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-australia","v/continent-australia",{type:"is-in"});

db.e.save("v/country-austria","v/continent-europe",{type:"is-in"});

db.e.save("v/country-bahamas","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-bahrain","v/continent-asia",{type:"is-in"});

db.e.save("v/country-bangladesh","v/continent-asia",{type:"is-in"});

db.e.save("v/country-barbados","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-belgium","v/continent-europe",{type:"is-in"});

db.e.save("v/country-bhutan","v/continent-asia",{type:"is-in"});

db.e.save("v/country-bolivia","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-bosnia-and-herzegovina","v/continent-europe",{type:"is-in"});

db.e.save("v/country-botswana","v/continent-africa",{type:"is-in"});

db.e.save("v/country-brazil","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-brunei","v/continent-asia",{type:"is-in"});

db.e.save("v/country-bulgaria","v/continent-europe",{type:"is-in"});

db.e.save("v/country-burkina-faso","v/continent-africa",{type:"is-in"});

db.e.save("v/country-burundi","v/continent-africa",{type:"is-in"});

db.e.save("v/country-cambodia","v/continent-asia",{type:"is-in"});

db.e.save("v/country-cameroon","v/continent-africa",{type:"is-in"});

db.e.save("v/country-canada","v/continent-north-america",{type:"is-in"});

db.e.save("v/country-chad","v/continent-africa",{type:"is-in"});

db.e.save("v/country-chile","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-colombia","v/continent-south-america",{type:"is-in"});

db.e.save("v/country-cote-d-ivoire","v/continent-africa",{type:"is-in"});

db.e.save("v/country-croatia","v/continent-europe",{type:"is-in"});

db.e.save("v/country-czech-republic","v/continent-europe",{type:"is-in"});

db.e.save("v/country-denmark","v/continent-europe",{type:"is-in"});

db.e.save("v/country-ecuador","v/continent-south-america",{type:"is-in"});

ExampleData

173

db.e.save("v/country-egypt","v/continent-africa",{type:"is-in"});

db.e.save("v/country-eritrea","v/continent-africa",{type:"is-in"});

db.e.save("v/country-finland","v/continent-europe",{type:"is-in"});

db.e.save("v/country-france","v/continent-europe",{type:"is-in"});

db.e.save("v/country-germany","v/continent-europe",{type:"is-in"});

db.e.save("v/country-people-s-republic-of-china","v/continent-asia",{type:"is-in"});

//edges:capital->country

db.e.save("v/capital-algiers","v/country-algeria",{type:"is-in"});

db.e.save("v/capital-andorra-la-vella","v/country-andorra",{type:"is-in"});

db.e.save("v/capital-asmara","v/country-eritrea",{type:"is-in"});

db.e.save("v/capital-bandar-seri-begawan","v/country-brunei",{type:"is-in"});

db.e.save("v/capital-beijing","v/country-people-s-republic-of-china",{type:"is-in"});

db.e.save("v/capital-berlin","v/country-germany",{type:"is-in"});

db.e.save("v/capital-bogota","v/country-colombia",{type:"is-in"});

db.e.save("v/capital-brasilia","v/country-brazil",{type:"is-in"});

db.e.save("v/capital-bridgetown","v/country-barbados",{type:"is-in"});

db.e.save("v/capital-brussels","v/country-belgium",{type:"is-in"});

db.e.save("v/capital-buenos-aires","v/country-argentina",{type:"is-in"});

db.e.save("v/capital-bujumbura","v/country-burundi",{type:"is-in"});

db.e.save("v/capital-cairo","v/country-egypt",{type:"is-in"});

db.e.save("v/capital-canberra","v/country-australia",{type:"is-in"});

db.e.save("v/capital-copenhagen","v/country-denmark",{type:"is-in"});

db.e.save("v/capital-dhaka","v/country-bangladesh",{type:"is-in"});

db.e.save("v/capital-gaborone","v/country-botswana",{type:"is-in"});

db.e.save("v/capital-helsinki","v/country-finland",{type:"is-in"});

db.e.save("v/capital-kabul","v/country-afghanistan",{type:"is-in"});

db.e.save("v/capital-la-paz","v/country-bolivia",{type:"is-in"});

db.e.save("v/capital-luanda","v/country-angola",{type:"is-in"});

db.e.save("v/capital-manama","v/country-bahrain",{type:"is-in"});

db.e.save("v/capital-nassau","v/country-bahamas",{type:"is-in"});

db.e.save("v/capital-n-djamena","v/country-chad",{type:"is-in"});

db.e.save("v/capital-ottawa","v/country-canada",{type:"is-in"});

db.e.save("v/capital-ouagadougou","v/country-burkina-faso",{type:"is-in"});

db.e.save("v/capital-paris","v/country-france",{type:"is-in"});

db.e.save("v/capital-phnom-penh","v/country-cambodia",{type:"is-in"});

db.e.save("v/capital-prague","v/country-czech-republic",{type:"is-in"});

db.e.save("v/capital-quito","v/country-ecuador",{type:"is-in"});

db.e.save("v/capital-saint-john-s","v/country-antigua-and-barbuda",{type:"is-in"});

db.e.save("v/capital-santiago","v/country-chile",{type:"is-in"});

db.e.save("v/capital-sarajevo","v/country-bosnia-and-herzegovina",{type:"is-in"});

db.e.save("v/capital-sofia","v/country-bulgaria",{type:"is-in"});

db.e.save("v/capital-thimphu","v/country-bhutan",{type:"is-in"});

db.e.save("v/capital-tirana","v/country-albania",{type:"is-in"});

db.e.save("v/capital-vienna","v/country-austria",{type:"is-in"});

db.e.save("v/capital-yamoussoukro","v/country-cote-d-ivoire",{type:"is-in"});

db.e.save("v/capital-yaounde","v/country-cameroon",{type:"is-in"});

db.e.save("v/capital-zagreb","v/country-croatia",{type:"is-in"});

ExampleData

174

Edges,Identifiers,HandlesThisisanintroductiontoArangoDB'sinterfaceforedges.Edgesmaybeusedingraphs.HereweworkwithedgesfromtheJavaScriptshellarangosh.ForotherlanguagesseethecorrespondinglanguageAPI.

Agraphdatamodelalwaysconsistsofatleasttwocollections:therelationsbetweenthenodesinthegraphsarestoredinan"edgescollection",thenodesinthegrapharestoredindocumentsinregularcollections.

EdgesinArangoDBarespecialdocuments.Inadditiontothesystemattributes_key,_idand_rev,theyhavetheattributes_fromand_to,whichcontaindocumenthandles,namelythestart-pointandtheend-pointoftheedge.

Example:

the"edge"collectionstorestheinformationthatacompany'sreceptionissub-unittotheservicesunitandtheservicesunitissub-unittotheCEO.Youwouldexpressthisrelationshipwiththe_fromand_toattributesthe"normal"collectionstoresallthepropertiesaboutthereception,e.g.that20peopleareworkingthereandtheroomnumberetc_fromisthedocumenthandleofthelinkedvertex(incomingrelation)_toisthedocumenthandleofthelinkedvertex(outgoingrelation)

Edgecollectionsarespecialcollectionsthatstoreedgedocuments.Edgedocumentsareconnectiondocumentsthatreferenceotherdocuments.Thetypeofacollectionmustbespecifiedwhenacollectioniscreatedandcannotbechangedafterwards.

Tochangeedgeendpointsyouwouldneedtoremoveolddocument/edgeandinsertnewone.Otherfieldscanbeupdatedasindefaultcollection.

WorkingwithEdges

Edgesarenormaldocumentsthatalwayscontaina_fromanda_toattribute.

WorkingwithEdges

175

DistributedIterativeGraphProcessing(Pregel)Distributedgraphprocessingenablesyoutodoonlineanalyticalprocessingdirectlyongraphsstoredintoarangodb.Thisisintendedtohelpyougainanalyticalinsightsonyourdata,withouthavingtouseexternalprocessingsytems.ExamplesofalgorithmstoexecutearePageRank,VertexCentrality,VertexCloseness,ConnectedComponents,CommunityDetection.Thissystemisnotusefulfortypicalonlinequeries,whereyoujustdoworkonasmallsetofvertices.ThesekindoftasksarebettersuitedforAQL.

TheprocessingsysteminsideArangoDBisbasedon:Pregel:ASystemforLarge-ScaleGraphProcessing–Malewiczetal.(Google)2010Thisconceptenablesustoperformdistributedgraphprocessing,withouttheneedfordistributedgloballocking.

PrerequisitesIfyouarerunningasingleArangoDBinstanceinsingle-servermode,therearenorequirementsregardingthemodelingofyourdata.Allyouneedisatleastonevertexcollectionandoneedgecollection.Notethattheperformancemaybebetter,ifthenumberofyourshards/collectionsmatchesthenumberofCPUcores.

WhenyouuseArangoDBCommunityeditioninclustermode,youmightneedtomodelyourcollectionsinacertainwaytoensurecorrectresults.Formoreinformationseethenextsection.

RequirementsforCollectionsinaCluster(NonSmartGraph)Toenableiterativegraphprocessingforyourdata,youwillneedtoensurethatyourvertexandedgecollectionsareshardedinaspecificway.

ThepregelcomputingmodelrequiresalledgestobepresentontheDBServerwherethevertexdocumentidentifiedbythe_fromvalueislocated.Thismeansthevertexcollectionsneedtobeshardedby'_key'andtheedgecollectionwillneedtobeshardedafteranattributewhichalwayscontainsthe'_key'ofthevertex.

Ourimplementationcurrentlyrequireseveryedgecollectiontobeshardedaftera"vertex"attributes,additionallyyouwillneedtospecifythekeydistributeShardsLikeandanequalnumberofshardsoneverycollection.OnlyiftheserequirementsaremetcanArangoDBplacetheedgesandverticescorrectly.

Forexampleyoumightcreateyourcollectionslikethis:

//Createmainvertexcollection:

db._create("vertices",{

shardKeys:['_key'],

numberOfShards:8

});

//Optionallycreatearbitraryadditionalvertexcollections

db._create("additonal",{

distributeShardsLike:"vertices",

numberOfShards:8

});

//Create(oneormore)edge-collections:

db._createEdgeCollection("edges",{

shardKeys:['vertex'],

distributeShardsLike:"vertices",

numberOfShards:8

});

Youwillneedtoensurethatedgedocumentscontainthepropervaluesintheirshardingattribute.Foravertexdocumentwiththefollowingcontent{_key:"A",value:0}thecorrespondingedgedocumentswouldhavelooklikethis:

{_from:"vertices/A",_to:"vertices/B",vertex:"A"}

{_from:"vertices/A",_to:"vertices/C",vertex:"A"}

{_from:"vertices/A",_to:"vertices/D",vertex:"A"}

...

Pregel

176

http://www.dcs.bbk.ac.uk/~dell/teaching/cc/paper/sigmod10/p135-malewicz.pdf

ThiswillensurethatoutgoingedgedocumentswillbeplacedonthesameDBServerasthevertex.Withoutthecorrectplacementoftheedges,thepregelgraphprocessingsystemwillnotworkcorrectly,becauseedgeswillnotloadcorrectly.

ArangoshAPI

StartinganAlgorithmExecution

ThepregelAPIisaccessiblethroughthe@arangodb/pregelpackage.Tostartanexecutionyouneedtospecifythealgorithmnameandthevertexandedgecollections.Alternativelyyoucanspecifyanamedgraph.Additionallyyoucanspecifycustomparameterswhichvaryforeachalgorithm.ThestartmethodwillalwaysauniqueIDwhichcanbeusedtointeractwiththealgorithmandlateron.

Thebelowversionofthestartmethodcanbeusedfornamedgraphs:

varpregel=require("@arangodb/pregel");

varparams={};

varexecution=pregel.start("<algorithm>","<yourgraph>",params);

Paramsneedstobeanobject,thevalidkeysarementionedbelowinthesectionAlgorithms

Alternativelyyoumightwanttospecifythevertexandedgecollectionsdirectly.Thecall-syntaxofthestart``methodchangesinthiscase.ThesecondargumentmustbeanobjectwiththekeysvertexCollectionsandedgeCollections`.


varparams={};

varexecution=pregel.start("<algorithm>",{vertexCollections:["vertices"],edgeCollections:["edges"]},{});

Thelastargumentisstilltheparameterobject.Seebelowforalistofalgorithmsandparameters.

StatusofanAlgorithmExecution

Thecodereturnedbythepregel.start(...)methodcanbeusedtotrackthestatusofyouralgorithm.

varexecution=pregel.start("sssp","demograph",{source:"vertices/V"});

varstatus=pregel.status(execution);

Theresultwilltellyouthecurrentstatusofthealgorithmexecution.Itwilltellyouthecurrentstateoftheexecution,thecurrentglobalsuperstep,theruntime,theglobalaggregatorvaluesaswellasthenumberofsendandreceivedmessages.

Validvaluesforthestatefieldinclude:

"running"algorithmisstillrunning"done":Theexecutionisdone,theresultmightnotbewrittenbackintothecollectionyet."canceled":Theexecutionwaspermanentlycanceled,eitherbytheuserorbyanerror."inerror":Theexeuctionisinanerrorstate.ThiscanbecausedbyprimaryDBServersbeingnotreachableorbeingnonresponsive.Theexecutionmightrecoverlater,orswitchto"canceled"ifitwasnotabletorecoversuccessfuly"recovering":Theexecutionisactivelyrecovering,willswitchbackto"running"iftherecoverywassuccessful

Theobjectreturnedbythestatusmethodmightforexamplelooksomethinglikethis:

{

"state":"running",

"gss":12,

"totalRuntime":123.23,

"aggregators":{

"converged":false,

"max":true,

"phase":2

},

"sendCount":3240364978,

"receivedCount":3240364975

}

Pregel

177

CancelinganExecution/Discardingresults

Tocancelanexecutionwhichisstillrunnning,anddiscardanyintermediareresultsyoucanusethecancelmethod.Thiswillimmediatlyfreeallmemorytakenupbytheexecution,andwillmakeyouloseallintermediarydata.

Youmightgetinconsistentresultsifyoucancelanexecutionwhileitisalreadyinit'sdonestate.Thedataiswrittenmulti-threadedintoallcollectionshardsatonce,thismeanstherearemultipletransactionssimultaniously.Atransactionmightalreadybecommitedwhenyoucanceltheexecutionjob,thereforeyoumightseetheresultinyourcollection.Thisdoesnotapplyifyouconfiguredtheexecutiontonotwritedataintothecollection.

//startasinglesourceshortestpathjob

varexecution=pregel.start("sssp","demograph",{source:"vertices/V"});

pregel.cancel(execution);

AQLintegrationArangoDBsupportsretrievingtemporarypregelresultsthroughtheArangoDBquerylanguage(AQL).Whenourgraphprocessingsubsystemfinishesexecutinganalgorithm,theresultcaneitherbewrittenbackintothedatabaseorkeptinmemory.InbothcasestheresultcanbequeriedviaAQL.Ifthedatawasnotwrittentothedatabasestoreitisonlyheldtemporarily,untiltheusercallsthecancelmethodForexampleausermightwanttoqueryonlynodeswiththemostrankfromtheresultsetofaPageRankexecution.

FORvINPREGEL_RESULT(<handle>)

FILTERv.value>=0.01

RETURNv._key

AvailableAlgorithmsThereareanumberofgeneralparameterswhichapplytoalmostallalgorithms:

store:Isperdefaulttrue,thepregelenginewillwriteresultsbacktothedatabase.ifthevalueisfalsethenyoucanquerytheresultsviaAQLk,seeAQLintegration.maxGSS:Maximumnumberofglobaliterationsforthisalgorithmparallelism:Numberofparellelthreadstouseperworker.Doesnotinfluencethenumberofthreadsusedtoload

orstoredatafromthedatabase(thisdependsonthenumberofshards).

async:Algorithmswichsupportasyncmode,willrunwithoutsynchronizedglobaliterations,mightleadtoperformanceincreasesifyouhaveloadimbalances.resultField:Mostalgorithmswillwritetheresultintothisfield

PageRank

PageRankisawellknownalgorithmtorankdocumentsinagraph.Thealgorithmwillrununtiltheexecutionconverges.Specifyacustomthresholdwiththeparameterthreshold,torunforafixednumberofiterationsusethemaxGSSparameter.


pregel.start("pagerank","graphname",{maxGSS:100,threshold:0.00000001})

Single-SourceShortestPath

Calculatesthedistanceofeachvertextoacertainshortestpath.Thealgorithmwillrununtilitconverges,theiterationsareboundbythediameter(thelongestshortestpath)ofyourgraph.


pregel.start("sssp","graphname",{source:"vertices/1337"})

ConnectedComponents

Pregel

178

Therearetwoalgorithmstofindconnectedcomponentsinagraph.Tofindweaklyconnectedcomponents(WCC)youcanusethealgorithmnamed"connectedcomponents",tofindstronglyconnectedcomponents(SCC)youcanusethealgorithmnamed"scc".BothalgorithmwillassignacomponentIDtoeachvertex.

Aweaklyconnectedcomponentsmeansthatthereexistapathfromeveryvertexpairinthatcomponent.WCCisaverysimpleandfastalgorithm,whichwillonlyworkcorrectlyonundirectedgraphs.Yourresultsondirectedgraphsmayvary,dependingonhowconnectedyourcomponentsare.

InthecaseofSCCacomponentmeanseveryvertexisreachablefromanyothervertexinthesamecomponent.ThealgorithmismorecomplexthantheWCCalgorithmandrequiresmoreRAM,becauseeachvertexneedstostoremuchmorestate.ConsiderusingWCCifyouthinkyourdatamaybesuitableforit.


//weaklyconnectedcomponents

pregel.start("connectedcomponents","graphname")

//stronglyconnectedcomponents

pregel.start("scc","graphname")

Hyperlink-InducedTopicSearch(HITS)

HITSisalinkanalysisalgorithmthatratesWebpages,developedbyJonKleinberg(Thealgorithmisalsoknownashubsandauthorities).

TheideabehindHubsandAuthoritiescomesfromthetypicalstructureoftheweb:Certainwebsitesknownashubs,serveaslargedirectoriesthatarenotactuallyauthoritativeontheinformationthattheyhold.Thesehubsareusedascompilationsofabroadcatalogofinformationthatleadsusersdirecttootherauthoritativewebpages.Thealgorithmassignseachvertextwoscores:Theauthority-scoreandthehub-score.Theauthorityscorerateshowmanygoodhubspointtoaparticularvertex(orwebpage),thehubscorerateshowgood(authoritative)theverticespointedtoare.Formoreseehttps://en.wikipedia.org/wiki/HITS_algorithm

Ourversionofthealgorithmconvergesafteracertainamountoftime.Theparameterthresholdcanbeusedtosetalimitfortheconvergence(measuredasmaximumabsolutedifferenceofthehubandauthorityscoresbetweenthecurrentandlastiteration)Whenyouspecifytheresultfieldname,thehubscorewillbestoredin"_hub"andtheauthorityscorein"_auth".Thealgorithmcanbeexecutedlikethis:


varhandle=pregel.start("hits","yourgraph",{threshold:0.00001,resultField:"score"});

VertexCentrality

Centralitymeasureshelpidentifythemostimportantverticesinagraph.Theycanbeusedinawiderangeofapplications:Forexampletheycanbeusedtoidentifyinfluencersinsocialnetworks,ormiddle-meninterroristnetworks.Therearevariousdefinitionsforcentrality,thesimplestonebeingthevertexdegree.Thesedefinitionswerenotdesignedwithscalabilityinmind.Itisprobablyimpossibletodiscoveranefficientalgorithmwhichcomputestheminadistributedway.Fortunatelytherearescalablesubstitutionsavailable,whichshouldbeequallyusableformostusecases.

Pregel

179

https://en.wikipedia.org/wiki/HITS_algorithm

EffectiveCloseness

Acommondefinitionsofcentralityistheclosenesscentrality(orcloseness).Theclosenessofavertexinagraphistheinverseaveragelengthoftheshortestpathbetweenthevertexandallothervertices.Forverticesx,yandshortestdistanced(y,x)itisdefinedas

EffectiveClosenessapproximatestheclosenessmeasure.Thealgorithmworksbyiterativelyestimatingthenumberofshortestpathspassingthrougheachvertex.Thescorewillapproximatesthetherealclosenessscore,sinceitisnotpossibletoactuallycountallshortestpathsduetothehorrendousO(n^2d)memoryrequirements.ThealgorithmisfromthepaperCentralitiesinLargeNetworks:AlgorithmsandObservations(UKanget.al.2011)*

ArangoDBsimplementationapproximatesthenumberofshortestpathineachiterationbyusingaHyperLogLogcounterwith64buckets.Thisshouldworkwellonlargegraphsandonsmalleronesaswell.ThememoryrequirementsshouldbeO(n*d)wherenisthenumberofverticesanddthediameterofyourgraph.Eachvertexwillstoreacounterforeachiterationofthealgorithm.Thealgorithmcanbeusedlikethis

constpregel=require("@arangodb/pregel");

consthandle=pregel.start("effectivecloseness","yourgraph",{resultField:"closeness"});

LineRank

Anothercommonmeasureisthebetweenness*centrality:Itmeasuresthenumberoftimesavertexispartofshortestpathsbetweenanypairsofvertices.Foravertexvbetweennessisdefinedas

Wheretheσrepresentsthenumberofshortestpathsbetweenxandy,andσ(v)representsthenumberofpathsalsopassingthroughavertexv.Byintuitionavertexwithhigherbetweenesscentralitywillhavemoreinformationpassingthroughit.

LineRankapproximatestherandomwalkbetweennessofeveryvertexinagraph.Thisistheprobabilitythatsomeonestartingonanarbitaryvertex,willvisitthisnodewhenherandomlychoosesedgestovisit.Thealgoruthmessentiallybuildsalinegraphoutofyourgraph(switchestheverticesandedges),andthencomputesascoresimilartoPageRank.Thiscanbeconsideredascalableequivalenttovertexbetweeness,whichcanbeexecuteddistributedlyinArangoDB.ThealgorithmisfromthepaperCentralitiesinLargeNetworks:AlgorithmsandObservations(UKanget.al.2011)

Pregel

180

https://en.wikipedia.org/wiki/Betweenness_centrality


consthandle=pregel.start("linerank","yourgraph",{"resultField":"rank"});

CommunityDetection

Graphsbasedonrealworldnetworksoftenhaveacommunitystructure.Thismeansitispossibletofindgroupsofverticessuchthateacheachvertexgroupisinternallymoredenselyconnectedthanoutsidethegroup.Thishasmanyapplicationswhenyouwanttoanalyzeyournetworks,forexampleSocialnetworksincludecommunitygroups(theoriginoftheterm,infact)basedoncommonlocation,interests,occupation,etc.

LabelPropagation

LabelPropagationcanbeusedtoimplementcommunitydetectiononlargegraphs.Theideaisthateachvertexshouldbeinthecommunitythatmostofhisneighboursarein.WeiterativelydeteminethisbyfirstassigningrandomCommunityID's.Theneachitertation,avertexwillsendit'scurrentcommunityIDtoallhisneighborvertices.TheneachvertexadoptsthecommunityIDhereceivedmostfrequentlyduringtheiteration.

Thealgorithmrunsuntilitconverges,whichlikelyneverreallyhappensonlargegraphs.Thereforeyouneedtospecifyamaximumiterationboundwhichsuitsyou.Thedefaultboundis500iterations,whichislikelytoolargeforyourapplication.Shouldworkbestonundirectedgraphs,resultsondirectedgraphsmightvarydependingonthedensityofyourgraph.


consthandle=pregel.start("labelpropagation","yourgraph",{maxGSS:100,resultField:"community"});

Speaker-ListenerLabelPropagation

TheSpeaker-listenerLabelPropagation(SLPA)canbeusedtoimplementcommunitydetection.Itworkssimilartothelabelpropagationalgorithm,butnoweverynodeadditionallyaccumulatesamemoryofobservedlabels(insteadofforgettingallbutonelabel).

Beforethealgorithmrun,everyvertexisinitializedwithanuniqueID(theinitialcommunitylabel).Duringtherunthreestepsareexecutedforeachvertex:

1. Currentvertexisthelistenerallotherverticesarespeakers2. Eachspeakersendsoutalabelfrommemory,wesendoutarandomlabelwithaprobabilityproportionaltothenumberoftimesthe

vertexobservedthelabel3. Thelistenerremembersoneofthelabels,wealwayschoosethemostfrequentlyobservedlabel


consthandle=pregel.start("slpa","yourgraph",{maxGSS:100,resultField:"community"});

YoucanalsoexecuteSLPAwiththemaxCommunitiesparametertolimitthenumberofouputcommunities.Internallythealgorithmwillstillkeepthememoryofalllabels,buttheoutputisreducedtojusthenmostfrequentlyobservedlabels.


consthandle=pregel.start("slpa","yourgraph",{maxGSS:100,resultField:"community",maxCommunities:1});

//checkthestatusperiodicallyforcompletion

pregel.status(handle);

Pregel

181

https://arxiv.org/pdf/1109.5720.pdf

FoxxTraditionally,server-sideprojectshavebeendevelopedasstandaloneapplicationsthatguidethecommunicationbetweentheclient-sidefrontendandthedatabasebackend.Thishasledtoapplicationsthatwereeitherdevelopedassinglemonolithsorthatduplicateddataaccessanddomainlogicacrossallservicesthathadtoaccessthedatabase.Additionally,toolstoabstractawaytheunderlyingdatabasecallscouldincuralotofnetworkoverheadwhenusingremotedatabaseswithoutcarefuloptimization.

ArangoDBallowsapplicationdeveloperstowritetheirdataaccessanddomainlogicasmicroservicesrunningdirectlywithinthedatabasewithnativeaccesstoin-memorydata.TheFoxxmicroserviceframeworkmakesiteasytoextendArangoDB'sownRESTAPIwithcustomHTTPendpointsusingmodernJavaScriptrunningonthesameV8engineyouknowfromNode.jsandtheGoogleChromewebbrowser.

Unliketraditionalapproachestostoringlogicinthedatabase(likestoredprocedures),thesemicroservicescanbewrittenasregularstructuredJavaScriptapplicationsthatcanbeeasilydistributedandversioncontrolled.Dependingonyourproject'sneedsFoxxcanbeusedtobuildanythingfromoptimizedRESTendpointsperformingcomplexdataaccesstoentirestandaloneapplicationsrunningdirectlyinsidethedatabase.

FoxxMicroservices

182

FoxxataglanceEachFoxxserviceisdefinedbyaJSONmanifestspecifyingtheentrypoint,anyscriptsdefinedbytheservice,possibleconfigurationoptionsandFoxxdependencies,aswellasothermetadata.Withinaservice,theseoptionsareexposedastheservicecontext.

AttheheartoftheFoxxframeworkliestheFoxxRouterwhichisusedtodefineHTTPendpoints.AservicecanaccessthedatabaseeitherdirectlyfromitscontextusingprefixedcollectionsortheArangoDBdatabaseAPI.

WhileFoxxisprimarilydesignedtobeusedtoaccessthedatabaseitself,ArangoDBalsoprovidesanAPItomakeHTTPrequeststoexternalservices.

Scriptscanbeusedtoperformone-offtasks,whichcanalsobescheduledtobeperformedasynchronouslyusingthebuilt-injobqueue.

Finally,FoxxservicescanbeinstalledandmanagedovertheWeb-UIorthroughArangoDBsHTTPAPI.

HowdoesitworkFoxxservicesconsistofJavaScriptcoderunningintheV8JavaScriptruntimeembeddedinsideArangoDB.EachserviceismountedineachavailableV8context(thenumberofcontextscanbeadjustedintheArangoDBconfiguration).Incomingrequestsaredistributedaccrossthesecontextsautomatically.

Ifyou'recomingfromanotherJavaScriptenvironmentlikeNode.jsthisissimilartorunningmultipleNode.jsprocessesbehindaloadbalancer:youshouldnotrelyonserver-sidestate(otherthanthedatabaseitself)betweendifferentrequestsasthereisnowayofmakingsureconsecutiverequestswillbehandledinthesamecontext.

BecausetheJavaScriptcodeisrunninginsidethedatabaseanotherdifferenceisthatallFoxxandArangoDBAPIsarepurelysynchronousandshouldbeconsideredblocking.Thisisespeciallyimportantfortransactions,whichinArangoDBcanexecutearbitrarycodebutmayhavetolockentirecollections(effectivelypreventinganydatatobewritten)untilthecodehascompleted.

Forinformationonhowthisaffectsinteroperabilitywiththird-partyJavaScriptmoduleswrittenforotherJavaScriptenvironmentsseethechapterondependencies.

DevelopmentmodeDevelopmentmodeallowsyoutomakechangestodeployedservicesin-placedirectlyonthedatabaseserver'sfilesystemwithoutdownloadingandre-uploadingtheservicebundle.Additionallyerrormessageswillcontainstacktraces.

YoucantoggledevelopmentmodeonandoffintheservicesettingstabofthewebinterfaceorusingtheHTTPAPI.Onceactivatedtheservice'sfilesystempathwillbeshownintheinfotab.

Onceenabledtheservice'ssourcefilesandmanifestwillbere-evaluated,andthesetupscript(ifpresent)re-executed,everytimearouteoftheserviceisaccessed,effectivelyre-deployingtheserviceoneveryrequest.Asthenameindicatesthisisintendedtobeusedstrictlyduringdevelopmentandismostdefinitelyabadideaonproductionservers.Theadditionalinformationexposedduringdevelopmentmodemayincludefilesystempathsandpartsoftheservice'ssourcecode.

Alsonotethatifyouareservingstaticfilesaspartofyourservice,accessingthesefilesfromabrowsermayalsotriggerare-deploymentoftheservice.Finally,makingHTTPrequeststoaservicerunningindevelopmentmodefromwithintheservice(i.e.usingthe@arangodb/requestmoduletoaccesstheserviceitself)isprobablynotagoodideaeither.

Bewareofdeletingthedatabasetheserviceisdeployedon:itwillerasethesourcefilesoftheservicealongwiththecollections.Youshouldbackupthecodeyouworkedonindevelopmentbeforedoingthattoavoidlosingyourprogress.

FoxxstoreTheFoxxstoreprovidesaccesstoanumberofready-to-useofficialandcommunity-maintainedFoxxservicesyoucaninstallwithasingleclick,includingexampleservicesandwrappersforexternalSaaStoolsliketransactionale-mailservices,bugloggersoranalyticstrackers.

YoucanfindtheFoxxstoreinthewebinterfacebyusingtheAddServicebuttonintheservicelist.

Ataglance

183

Cluster-Foxx

WhenrunningArangoDBinaclustertheFoxxserviceswillrunoneachcoordinator.Installing,upgradinganduninstallingservicesonanycoordinatorwillautomaticallydistributetheservicetotheothercoordinators,makingdeploymentsaseasyasinsingle-servermode.However,thismeanstherearesomelimitations:

Youshouldavoidanykindoffilesystemstatebeyondthedeployedservicebundleitself.Don'twritedatatothefilesystemorencodeanyexpectationsofthefilesystemstateotherthanthefilesintheservicefolderthatwereinstalledaspartoftheservice(e.g.fileuploadsorcustomlogfiles).

Additionally,thedevelopmentmodewillleadtoaninconsistentstateoftheclusteruntilitisdisabled.WhileaserviceisrunningindevelopmentmodeyoucanmakechangestotheserviceonthefilesystemofanycoordinatorandseethemreflectedinrealtimejustlikewhenrunningArangoDBasasingleserver.Howeverthechangesmadeononecoordinatorwillnotbereflectedacrossothercoordinatorsuntilthedevelopmentmodeisdisabled.Whendisablingthedevelopmentmodeforaservice,thecoordinatorwillcreateanewbundleanddistributeitacrosstheservicelikeamanualupgradeoftheservice.

Forthesereasonsitisstronglyrecommendednottousedevelopmentmodeinaclusterwithmultiplecoordinatorsunlessyouaresurethatnorequestsorchangeswillbemadetoothercoordinatorswhileyouaremodifyingtheservice.Usingdevelopmentmodeinaproductionclusterisextremelyunsafeandhighlydiscouraged.

Ataglance

184

GettingStartedWe'regoingtostartwithanemptyfolder.Thiswillbetherootfolderofourservices.Youcannameitsomethingcleverbutforthecourseofthisguidewe'llassumeit'scalledthenameofyourservice:getting-started.

Firstweneedtocreateamanifest.Createanewfilecalledmanifest.jsonandaddthefollowingcontent:

{

"engines":{

"arangodb":"^3.0.0"

}

}

ThisjusttellsArangoDBtheserviceiscompatiblewithversions3.0.0andlater(allthewayuptobutnotincluding4.0.0),allowingolderversionsofArangoDBtounderstandthatthisservicelikelywon'tworkforthemandnewerversionswhatbehaviortoemulateshouldtheystillsupportit.

Thelittlehattotheleftoftheversionnumberisnotatypo,it'scalleda"caret"andindicatestheversionrange.Foxxusessemanticversioning(alsocalled"semver")formostofitsversionhandling.Youcanfindoutmoreabouthowsemverworksattheofficialsemverwebsite.

Nextwe'llneedtospecifyanentrypointtoourservice.ThisistheJavaScriptfilethatwillbeexecutedtodefineourservice'sHTTPendpoints.Wecandothisbyaddinga"main"fieldtoourmanifest:

{

"engines":{

"arangodb":"^3.0.0"

},

"main":"index.js"

}

That'sallweneedinourmanifestfornow,solet'snextcreatetheindex.jsfile:

'usestrict';

constcreateRouter=require('@arangodb/foxx/router');

constrouter=createRouter();

module.context.use(router);

Thefirstlinecausesourfiletobeinterpretedusingstrictmode.AllexamplesintheArangoDBdocumentationassumestrictmode,soyoumightwanttofamiliarizeyourselfwithitifyouhaven'tencountereditbefore.

Thesecondlineimportsthe@arangodb/foxx/routermodulewhichprovidesafunctionforcreatingnewFoxxrouters.We'reusingthisfunctiontocreateanewrouterobjectwhichwe'llbeusingforourservice.

Themodule.contextistheso-calledFoxxcontextorservicecontext.ThisvariableisavailableinallfilesthatarepartofyourFoxxserviceandprovidesaccesstoFoxxAPIsspecifictothecurrentservice,liketheusemethod,whichtellsFoxxtomounttherouterinthisservice(andtoexposeitsroutestoHTTP).

Nextlet'sdefinearoutethatprintsagenericgreeting:

//continued

router.get('/hello-world',function(req,res){

res.send('HelloWorld!');

})

.response(['text/plain'],'Agenericgreeting.')

.summary('Genericgreeting')

.description('Printsagenericgreeting.');

Therouterprovidesthemethodsget,post,etccorrespondingtoeachHTTPverbaswellasthecatch-allall.ThesemethodsindicatethatthegivenrouteshouldbeusedtohandleincomingrequestswiththegivenHTTPverb(oranymethodwhenusingall).

Gettingstarted

185

http://semver.org

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode

Thesemethodstakeanoptionalpath(ifomitted,itdefaultsto"/")aswellasarequesthandler,whichisafunctiontakingthereq(request)andres(response)objectstohandletheincomingrequestandgeneratetheoutgoingresponse.IfyouhaveusedtheexpressframeworkinNode.js,youmayalreadybefamiliarwithhowthisworks,otherwisecheckoutthechapteronroutes.

Theobjectreturnedbytherouter'smethodsprovidesadditionalmethodstoattachmetadataandvalidationtotheroute.We'reusingsummaryanddescriptiontodocumentwhattheroutedoes--thesearen'tstrictlynecessarybutgiveussomeniceauto-generateddocumentation.Theresponsemethodletsusadditionallydocumenttheresponsecontenttypeandwhattheresponsebodywillrepresent.

Tryitout

AtthispointyoucanuploadtheservicefolderasaziparchivefromthewebinterfaceusingtheServicestab.

ClickAddServicethenpicktheZipoptioninthedialog.Youwillneedtoprovideamountpath,whichistheURLprefixatwhichtheservicewillbemounted(e.g./getting-started).

Onceyouhavepickedtheziparchiveusingthefilepicker,theuploadshouldbeginimmediatelyandyourserviceshouldbeinstalled.OtherwisepresstheInstallbuttonandwaitforthedialogtodisappearandtheservicetoshowupintheservicelist.

Clickanywhereonthecardwithyourmountpathonthelabeltoopentheservice'sdetails.

IntheAPIdocumentationyoushouldseetheroutewedefinedearlier(/hello-world)withthewordGETnexttoitindicatingtheHTTPmethoditsupportsandthesummaryweprovidedontheright.Byclickingontheroute'spathyoucanopenthedocumentationfortheroute.

Notethatthedescriptionweprovidedappearsinthegenerateddocumentationaswellasthedescriptionweaddedtotheresponse(whichshouldcorrectlyindicatethecontenttypetext/plain,i.e.plaintext).

ClicktheTryitout!buttontosendarequesttotherouteandyoushouldseeanexamplerequestwiththeservice'sresponse:"HelloWorld!".

Congratulations!Youhavejustcreated,installedandusedyourfirstFoxxservice.

Parametervalidation

Let'saddanotherroutethatprovidesamorepersonalizedgreeting:

//continued

constjoi=require('joi');

router.get('/hello/:name',function(req,res){

res.send(`Hello${req.pathParams.name}`);

})

.pathParam('name',joi.string().required(),'Nametogreet.')

.response(['text/plain'],'Apersonalizedgreeting.')

.summary('Personalizedgreeting')

.description('Printsapersonalizedgreeting.');

ThefirstlineimportsthejoimodulefromnpmwhichcomesbundledwithArangoDB.JoiisavalidationlibrarythatisusedthroughoutFoxxtodefineschemasandparametertypes.

Note:Youcanbundleyourownmodulesfromnpmbyinstallingtheminyourservicefolderandmakingsurethenode_modulesfolderisincludedinyourziparchive.Formoreinformationseethesectiononmoduledependenciesinthechapterondependencies.

ThepathParammethodallowsustospecifyparametersweareexpectinginthepath.Thefirstargumentcorrespondstotheparameternameinthepath,thesecondargumentisajoischematheparameterisexpectedtomatchandthefinalargumentservestodescribetheparameterintheAPIdocumentation.

ThepathparametersareaccessiblefromthepathParamspropertyoftherequestobject.We'reusingatemplatestringtogeneratetheserver'sresponsecontainingtheparameter'svalue.

Gettingstarted

186

https://www.npmjs.com/package/joi

NotethatrouteswithpathparametersthatfailtovalidatefortherequestURLwillbeskippedasiftheywouldn'texist.Thisallowsyoutodefinemultipleroutesthatareonlydistinguishedbytheschemasoftheirpathparameters(e.g.aroutetakingonlynumericparametersandonetakinganystringasafallback).

Let'stakethisfurtherandcreatearoutethattakesaJSONrequestbody:

//continued

router.post('/sum',function(req,res){

constvalues=req.body.values;

res.send({

result:values.reduce(function(a,b){

returna+b;

},0)

});

})

.body(joi.object({

values:joi.array().items(joi.number().required()).required()

}).required(),'Valuestoaddtogether.')

.response(joi.object({

result:joi.number().required()

}).required(),'Sumoftheinputvalues.')

.summary('Addupnumbers')

.description('Calculatesthesumofanarrayofnumbervalues.');

Notethatweusedposttodefinethisrouteinsteadofget(whichdoesnotsupportrequestbodies).TryingtosendaGETrequesttothisroute'sURL(intheabsenceofagetrouteforthesamepath)willresultinFoxxrespondingwithanappropriateerrorresponse,indicatingthesupportedHTTPmethods.

AsthisroutenotonlyexpectsaJSONobjectasinputbutalsorespondswithaJSONobjectasoutputweneedtodefinetwoschemas.Wedon'tstrictlyneedaresponseschemabutithelpsdocumentingwhattherouteshouldbeexpectedtorespondwithandwillshowupintheAPIdocumentation.

Becausewe'repassingaschematotheresponsemethodwedon'tneedtoexplicitlytellFoxxwearesendingaJSONresponse.ThepresenceofaschemaintheabsenceofacontenttypealwaysimplieswewantJSON.Thoughwecouldjustadd["application/json"]asanadditionalargumentaftertheschemaifwewantedtomakethismoreexplicit.

Thebodymethodworksthesamewayastheresponsemethodexcepttheschemawillbeusedtovalidatetherequestbody.Iftherequestbodycan'tbeparsedasJSONordoesn'tmatchtheschema,Foxxwillrejecttherequestwithanappropriateerrorresponse.

Creatingcollections

TherealpowerofFoxxcomesfrominteractingwiththedatabaseitself.Inordertobeabletouseacollectionfromwithinourservice,weshouldfirstmakesurethatthecollectionactuallyexists.Therightplacetocreatecollectionsyourserviceisgoingtouseisinasetupscript,whichFoxxwillexecuteforyouwheninstallingorupdatingtheservice.

Firstcreateanewfoldercalled"scripts"intheservicefolder,whichwillbewhereourscriptsaregoingtolive.Forsimplicity'ssake,oursetupscriptwillliveinafilecalledsetup.jsinsidethatfolder:

//continued

'usestrict';

constdb=require('@arangodb').db;

constcollectionName='myFoxxCollection';

if(!db._collection(collectionName)){

db._createDocumentCollection(collectionName);

}

Thescriptusesthedbobjectfromthe@arangodbmodule,whichletsusinteractwiththedatabasetheFoxxservicewasinstalledinandthecollectionsinsidethatdatabase.Becausethescriptmaybeexecutedmultipletimes(i.e.wheneverweupdatetheserviceorwhentheserverisrestarted)weneedtomakesurewedon'taccidentallytrytocreatethesamecollectiontwice(whichwouldresultinanexception);wedothatbyfirstcheckingwhetheritalreadyexistsbeforecreatingit.

Gettingstarted

187

The_collectionmethodlooksupacollectionbynameandreturnsnullifnocollectionwiththatnamewasfound.The_createDocumentCollectionmethodcreatesanewdocumentcollectionbyname(_createEdgeCollectionalsoexistsandworksanalogouslyforedgecollections).

Note:Becausewehavehardcodedthecollectionname,multiplecopiesoftheserviceinstalledalongsideeachotherinthesamedatabasewillsharethesamecollection.Becausethismaynotalwaysbewhatyouwant,theFoxxcontextalsoprovidesthecollectionNamemethodwhichappliesamountpointspecificprefixtoanygivencollectionnametomakeituniquetotheservice.Italsoprovidesthecollectionmethod,whichbehavesalmostexactlylikedb._collectionexceptitalsoappliestheprefixbeforelookingthecollectionup.

Nextweneedtotellourserviceaboutthescriptbyaddingittothemanifestfile:

{

"engines":{

"arangodb":"^3.0.0"

},

"main":"index.js",

"scripts":{

"setup":"scripts/setup.js"

}

}

Theonlythingthathaschangedisthatweaddeda"scripts"fieldspecifyingthepathofthesetupscriptwejustwrote.

Gobacktothewebinterfaceandupdatetheservicewithournewcode,thenchecktheCollectionstab.Ifeverythingworkedright,youshouldseeanewcollectioncalled"myFoxxCollection".

AccessingcollectionsLet'sexpandourservicebyaddingafewmoreroutestoourindex.js:

//continued


consterrors=require('@arangodb').errors;

constfoxxColl=db._collection('myFoxxCollection');

constDOC_NOT_FOUND=errors.ERROR_ARANGO_DOCUMENT_NOT_FOUND.code;

router.post('/entries',function(req,res){

constdata=req.body;

constmeta=foxxColl.save(req.body);

res.send(Object.assign(data,meta));

})

.body(joi.object().required(),'Entrytostoreinthecollection.')

.response(joi.object().required(),'Entrystoredinthecollection.')

.summary('Storeanentry')

.description('Storesanentryinthe"myFoxxCollection"collection.');

router.get('/entries/:key',function(req,res){

try{

constdata=foxxColl.document(req.pathParams.key);

res.send(data)

}catch(e){

if(!e.isArangoError||e.errorNum!==DOC_NOT_FOUND){

throwe;

}

res.throw(404,'Theentrydoesnotexist',e);

}

})

.pathParam('key',joi.string().required(),'Keyoftheentry.')

.response(joi.object().required(),'Entrystoredinthecollection.')

.summary('Retrieveanentry')

.description('Retrievesanentryfromthe"myFoxxCollection"collectionbykey.');

We'reusingthesaveanddocumentmethodsofthecollectionobjecttostoreandretrievedocumentsinthecollectionwecreatedinoursetupscript.Becausewedon'tcarewhatthedocumentslooklikeweallowanyattributesontherequestbodyandjustacceptanobject.

Gettingstarted

188

BecausethekeywillbeautomaticallygeneratedbyArangoDBwhenonewasn'tspecifiedintherequestbody,we'reusingObject.assigntoapplytheattributesofthemetadataobjectreturnedbythesavemethodtothedocumentbeforereturningitfromourfirstroute.

Thedocumentmethodreturnsadocumentinacollectionbyits_keyor_id.HoweverwhennomatchingdocumentexistsitthrowsanArangoErrorexception.BecausewewanttoprovideamoredescriptiveerrormessagethanArangoDBdoesoutofthebox,weneedtohandlethaterrorexplicitly.

AllArangoErrorexceptionshaveatruthyattributeisArangoErrorthathelpsyourecognizingtheseerrorswithouthavingtoworryaboutinstanceofchecks.TheyalsoprovideanerrorNumandanerrorMessage.Ifyouwanttocheckforspecificerrorsyoucanjustimporttheerrorsobjectfromthe@arangodbmoduleinsteadofhavingtomemorizenumericerrorcodes.

Insteadofdefiningourownresponselogicfortheerrorcasewejustuseres.throw,whichmakestheresponseobjectthrowanexceptionFoxxcanrecognizeandconverttotheappropriateserverresponse.WealsopassalongtheexceptionitselfsoFoxxcanprovidemorediagnosticinformationwhenwewantitto.

Wecouldextendthepostroutetosupportarraysofobjectsaswell,eachobjectfollowingacertainschema:

//storeschemainvariabletomakeitre-usable,see.body()

constdocSchema=joi.object().required().keys({

name:joi.string().required(),

age:joi.number().required()

}).unknown();//allowadditionalattributes

router.post('/entries',function(req,res){

constmultiple=Array.isArray(req.body);

constbody=multiple?req.body:[req.body];

letdata=[];

for(vardocofbody){

constmeta=foxxColl.save(doc);

data.push(Object.assign(doc,meta));

}

res.send(multiple?data:data[0]);

})

.body(joi.alternatives().try(

docSchema,

joi.array().items(docSchema)

),'Entryorentriestostoreinthecollection.')

.response(joi.alternatives().try(

joi.object().required(),

joi.array().items(joi.object().required())

),'Entryorentriesstoredinthecollection.')

.summary('Storeentryorentries')

.description('Storeasingleentryormultipleentriesinthe"myFoxxCollection"collection.');

WritingdatabasequeriesStoringandretrievingentriesisfine,butrightnowwehavetomemorizeeachkeywhenwecreateanentry.Let'saddaroutethatgivesusalistofthekeysofallentriessowecanusethosetolookanentryupindetail.

ThenaïveapproachwouldbetousethetoArray()methodtoconverttheentirecollectiontoanarrayandjustreturnthat.Butwe'reonlyinterestedinthekeysandtheremightpotentiallybesomanyentriesthatfirstretrievingeverysingledocumentmightgetunwieldy.Let'swriteashortAQLquerytodothisinstead:

//continued

constaql=require('@arangodb').aql;

router.get('/entries',function(req,res){

constkeys=db._query(aql`

FORentryIN${foxxColl}

RETURNentry._key

`);

res.send(keys);

})

.response(joi.array().items(

joi.string().required()

Gettingstarted

189

).required(),'Listofentrykeys.')

.summary('Listentrykeys')

.description('Assemblesalistofkeysofentriesinthecollection.');

Herewe'reusingtwonewthings:

The_querymethodexecutesanAQLqueryintheactivedatabase.

Theaqltemplatestringhandlerallowsustowritemulti-lineAQLqueriesandalsohandlesqueryparametersandcollectionnames.InsteadofhardcodingthenameofthecollectionwewanttouseinthequerywecansimplyreferencethefoxxCollvariablewedefinedearlier--itrecognizesthevalueasanArangoDBcollectionobjectandknowswearespecifyingacollectionratherthanaregularvalueeventhoughAQLdistinguishesbetweenthetwo.

Note:Ifyouaren'tusedtoJavaScripttemplatestringsandtemplatestringhandlersjustthinkofaqlasafunctionthatreceivesthemultilinestringsplitatevery${}expressionaswellasanarrayofthevaluesofthoseexpressions--that'sactuallyallthereistoit.

Alternatively,here'saversionwithouttemplatestrings(noticehowmuchcleanertheaqlversionwillbeincomparisonwhenyouhavemultiplevariables):

constkeys=db._query(

'FORentryIN@@collRETURNentry._key',

{'@coll':foxxColl}

);

Nextsteps

YounowknowhowtocreateaFoxxservicefromscratch,howtohandleuserinputandhowtoaccessthedatabasefromwithinyourFoxxservicetostore,retrieveandquerydatayoustoreinsideArangoDB.ThisshouldallowyoutobuildmeaningfulAPIsforyourownapplicationsbuttherearemanymorethingsyoucandowithFoxx:

Needtogofaster?Turnondevelopmentmodeandhackonyourcoderightontheserver.

Concernedaboutsecurity?Youcouldaddauthenticationtoyourservicetoprotectaccesstothedatabeforeitevenleavesthedatabase.

Writingasinglepageapp?YoucouldstoresomebasicassetsrightinsideyourFoxxservice.

Needtointegrateexternalservices?YoucanmakeHTTPrequestsfrominsideFoxxandusequeuedjobstoperformthatworkinthebackground.

Tiredofreinventingthewheel?Learnaboutdependencies.

Everythingbroken?Youcanwriteteststomakesureyourlogicremainssound.

Gettingstarted

190

ManifestfilesEveryservicecomeswithamanifest.jsonfileprovidingmetadata.Thefollowingfieldsareallowedinmanifests:

configuration:Object(optional)

Anobjectdefiningtheconfigurationoptionsthisservicerequires.

defaultDocument:string(optional)

Ifspecified,the/(root)routeoftheservicewillautomaticallyredirecttothegivenrelativepath,e.g.:

"defaultDocument":"index.html"

ThiswouldhavethesameeffectascreatingthefollowingrouteinJavaScript:


constindexRouter=createRouter();

indexRouter.all('/',function(req,res){

res.redirect('index.html');

});

module.context.use(indexRouter);

Note:Asof3.0.0thisfieldcansafelybeomitted;thevaluenolongerdefaultsto"index.html".

dependencies:Object(optional)andprovides:Object(optional)

Objectsspecifyingotherservicesthisservicehasasdependenciesandwhatdependenciesitcanprovidetootherservices.

engines:Object(optional)

AnobjectindicatingthesemanticversionrangesofArangoDB(orcompatibleenvironments)theservicewillbecompatiblewith,e.g.:

"engines":{

"arangodb":"^3.0.0"

}

ThisshouldcorrectlyindicatetheminimumversionofArangoDBtheservicehasbeentestedagainst.FoxxmaintainsastrictsemanticversioningpolicyasofArangoDB3.0.0soitisgenerallysafetousesemverranges(e.g. 3.0.0tomatchanyversiongreaterorequalto3.0.0andbelow4.0.0)formaximumcompatibility.

files:Object(optional)

Anobjectdefiningfileassetsservedbythisservice.

lib:string(Default:".")

TherelativepathtotheFoxxJavaScriptfilesintheservice,e.g.:

"lib":"lib"

Thiswouldresultinthemainentrypoint(seebelow)andotherJavaScriptpathsbeingresolvedasrelativetothelibfolderinsidetheservicefolder.

main:string(optional)

Therelativepathtothemainentrypointofthisservice(relativetolib,seeabove),e.g.:

"main":"index.js"

ThiswouldresultinFoxxloadingandexecutingthefileindex.jswhentheserviceismountedorstarted.

Servicemanifest

191

http://semver.org

Note:whileitistechnicallypossibletoomitthisfield,youwilllikelywanttoprovideanentrypointtoyourserviceasthisistheonlywaytoexposeHTTProutesorexportaJavaScriptAPI.

scripts:Object(optional)

Anobjectdefiningnamedscriptsprovidedbythisservice,whichcaneitherbeuseddirectlyorasqueuedjobsbyotherservices.

tests:stringorArray<string>(optional)

ApathorlistofpathsofJavaScripttestsprovidedforthisservice.

Additionallymanifestscanprovidethefollowingmetadata:

author:string(optional)

Thefullnameoftheauthoroftheservice(i.e.you).Thiswillbeshowninthewebinterface.

contributors:Array<string>(optional)

Alistofnamesofpeoplethathavecontributedtothedevelopmentoftheserviceinsomeway.Thiswillbeshowninthewebinterface.

description:string(optional)

Ahuman-readabledescriptionoftheservice.Thiswillbeshowninthewebinterface.

keywords:Array<string>(optional)

Alistofkeywordsthathelpcategorizethisservice.ThisisusedbytheFoxxStoreinstallerstoorganizeservices.

license:string(optional)

Astringidentifyingthelicenseunderwhichtheserviceispublished,ideallyintheformofanSPDXlicenseidentifier.Thiswillbeshowninthewebinterface.

name:string(optional)

ThenameoftheFoxxservice.AllowedcharactersareA-Z,0-9,theASCIIhyphen(-)andunderscore(_)characters.Thenamemustnotstartwithanumber.Thiswillbeshowninthewebinterface.

thumbnail:string(optional)

Thefilenameofathumbnailthatwillbeusedalongsidetheserviceinthewebinterface.ThisshouldbeaJPEGorPNGimagethatlooksgoodatsizes50x50and160x160.

version:string(optional)

TheversionnumberoftheFoxxservice.Theversionnumbermustfollowthesemanticversioningformat.Thiswillbeshowninthewebinterface.

Examples

{

"name":"example-foxx-service",

"version":"3.0.0-dev",

"license":"MIT",

"description":"Anexampleservicewitharelativelyfull-featuredmanifest.",

"thumbnail":"foxx-icon.png",

"keywords":["demo","service"],

"author":"ArangoDBGmbH",

"contributors":[

"AlanPlum<[email protected]>"

],

"lib":"dist",

"main":"entry.js",

"defaultDocument":"welcome.html",

"engines":{

"arangodb":"^3.0.0"

},

"files":{

Servicemanifest

192

https://spdx.org/licenses

http://semver.org

"welcome.html":"assets/index.html",

"hello.jpg":"assets/hello.jpg"

"world.jpg":{

"path":"assets/world.jpg",

"type":"image/jpeg",

"gzip":false

}

},

"tests":"dist/**.spec.js"

}

Servicemanifest

193

FoxxservicecontextTheservicecontextprovidesaccesstomethodsandattributesthatarespecifictoagivenservice.InaFoxxservicethecontextisgenerallyavailableasthemodule.contextvariable.Withinarouter'srequesthandlertherequestandresponseobjects'contextattributealsoprovideaccesstothecontextoftheservicetheroutewasmountedin(whichmaybedifferentfromtheonetheroutehandlerwasdefinedin).

Examples

//inservice/my-foxx-1



//Seethechapterondependenciesformoreinfoon

//howexportsanddependenciesworkacrossservices

module.exports={routes:router};

router.get(function(req,res){

module.context.mount==='/my-foxx-1';

req.context.mount==='/my-foxx-2';

res.write('Hellofrommy-foxx-1');

});

//inservice/my-foxx-2


constrouter2=createRouter();

module.context.use(router2);

router2.post(function(req,res){

module.context.mount==='/my-foxx-2';

req.context.mount==='/my-foxx-2';

res.write('Hellofrommy-foxx-2');

});

constrouter1=module.context.dependencies.myFoxx1.routes;

module.context.use(router1);

Theservicecontextspecifiesthefollowingproperties:

argv:any

Anyargumentspassedinifthecurrentfilewasexecutedasascriptorqueuedjob.

basePath:string

Thefilesystempathoftheservice,i.e.thefolderinwhichtheservicewasinstalledtobyArangoDB.

baseUrl:string

ThebaseURLoftheservice,relativetotheArangoDBserver,e.g./_db/_system/my-foxx.

collectionPrefix:string

TheprefixthatwillbeusedbycollectionandcollectionNametoderivethenamesofservice-specificcollections.Thisisderivedfromtheservice'smountpoint,e.g./my-foxxbecomesmy_foxx.

configuration:Object

Configurationoptionsfortheservice.

dependencies:Object

Configureddependenciesfortheservice.

isDevelopment:boolean

Indicateswhethertheserviceisrunningindevelopmentmode.

Servicecontext

194

isProduction:boolean

TheinverseofisDevelopment.

manifest:Object

Theparsedmanifestfileoftheservice.

mount:string

Themountpointoftheservice,e.g./my-foxx.

apiDocumentationmodule.context.apiDocumentation([options]):Function

DEPRECATED

CreatesarequesthandlerthatservestheAPIdocumentation.

Note:ThismethodhasbeendeprecatedinArangoDB3.1andreplacedwiththemorestraightforwardcreateDocumentationRoutermethodprovidingthesamefunctionality.

Arguments

SeecreateDocumentationRouterbelow.

Examples

//ServetheAPIdocsforthecurrentservice

router.get('/docs/*',module.context.apiDocumentation());

//Notethatthepathmustendwithawildcard

//andtheroutemustuseHTTPGET.

createDocumentationRoutermodule.context.createDocumentationRouter([options]):Router

CreatesarouterthatservestheAPIdocumentation.

Note:Theroutercanbemountedlikeanyotherchildrouter(seeexamplesbelow).

Arguments

options:Object(optional)

Anobjectwithanyofthefollowingproperties:

mount:string(Default:module.context.mount)

Themountpathoftheservicetoservethedocumentationof.

indexFile:string(Default:"index.html")

FilenameoftheHTMLfileservingtheAPIdocumentation.

swaggerRoot:string(optional)

FullpathofthefoldercontainingtheSwaggerassetsandtheindexFile.DefaultstotheSwaggerassetsusedbythewebinterface.

before:Function(optional)

Afunctionthatwillbeexecutedbeforearequestishandled.

Ifthefunctionreturnsfalsetherequestwillnotbeprocessedanyfurther.

Ifthefunctionreturnsanobject,itsattributeswillbeusedtooverridetheoptionsforthecurrentrequest.

Servicecontext

195

Anyotherreturnvaluewillbeignored.

Ifoptionsisafunctionitwillbeusedasthebeforeoption.

IfoptionsisastringitwillbeusedastheswaggerRootoption.

ReturnsaFoxxrouter.

Examples

//ServetheAPIdocsforthecurrentservice

router.use('/docs',module.context.createDocumentationRouter());

//--or--

//ServetheAPIdocsfortheservicetherouterismountedin

router.use('/docs',module.context.createDocumentationRouter(function(req){

return{mount:req.context.mount};

}));

//--or--

//ServetheAPIdocsonlyforusersauthenticatedwithArangoDB

router.use('/docs',module.context.createDocumentationRouter(function(req,res){

if(req.suffix==='swagger.json'&&!req.arangoUser){

res.throw(401,'Notauthenticated');

}

}));

collectionmodule.context.collection(name):ArangoCollection|null

PassesthegivennametocollectionName,thenlooksupthecollectionwiththeprefixedname.

Arguments

name:string

Unprefixednameoftheservice-specificcollection.

Returnsacollectionornullifnocollectionwiththeprefixednameexists.

collectionNamemodule.context.collectionName(name):string

PrefixesthegivennamewiththecollectionPrefixforthisservice.

Arguments

name:string

Unprefixednameoftheservice-specificcollection.

Returnstheprefixedname.

Examples

module.context.mount==='/my-foxx'

module.context.collectionName('doodads')==='my_foxx_doodads'

filemodule.context.file(name,[encoding]):Buffer|string

PassesthegivennametofileName,thenloadsthefilewiththeresultingname.

Servicecontext

196

Arguments

name:string

Nameofthefiletoload,relativetothecurrentservice.

encoding:string(optional)

Encodingofthefile,e.g.utf-8.Ifomittedthefilewillbeloadedasarawbufferinsteadofastring.

Returnsthefile'scontents.

fileNamemodule.context.fileName(name):string

Resolvesthegivenfilenamerelativetothecurrentservice.

Arguments

name:string

Nameofthefile,relativetothecurrentservice.

Returnstheabsolutefilepath.

usemodule.context.use([path],router):Endpoint

Mountsagivenrouterontheservicetoexposetherouter'sroutesontheservice'smountpoint.

Arguments

path:string(Default:"/")

Pathtomounttherouterat,relativetotheservice'smountpoint.

router:Router|Middleware

Arouterormiddlewaretomount.

ReturnsanEndpointforthegivenrouterormiddleware.

Note:Mountingservicesatruntime(e.g.withinrequesthandlersorqueuedjobs)isnotsupported.

Servicecontext

197

FoxxconfigurationFoxxservicescandefineconfigurationparameterstomakethemmorere-usable.

Theconfigurationobjectmapsnamestoconfigurationparameters:

Thekeyisthenameunderwhichtheparameterwillbeavailableontheservicecontext'sconfigurationproperty.

Thevalueisaparameterdefinition.

Theparameterdefinitioncanhavethefollowingproperties:

description:string

Humanreadabledescriptionoftheparameter.

type:string(Default:"string")

Typeoftheconfigurationparameter.Supportedvaluesare:

"integer"or"int":anyfiniteintegernumber.

"boolean"or"bool":thevaluestrueorfalse.

"number":anyfinitedecimalorintegernumber.

"string":anystringvalue.

"json":anywell-formedJSONvalue.

"password":likestringbutwillbedisplayedasamaskedinputfieldinthewebfrontend.

default:any

Defaultvalueoftheconfigurationparameter.

required:(Default:true)

Whethertheparameterisrequired.

Iftheconfigurationhasparametersthatdonotspecifyadefaultvalue,youneedtoconfiguretheservicebeforeitbecomesactive.InthemeantimeafallbackservicelicationwillbemountedthatrespondstoallrequestswithaHTTP500statuscodeindicatingaserver-sideerror.

TheconfigurationparametersofamountedservicecanbeadjustedfromthewebinterfacebyclickingtheConfigurationbuttonintheservicedetails.

Examples

"configuration":{

"currency":{

"description":"Currencysymboltouseforpricesintheshop.",

"default":"$",

"type":"string"

},

"secretKey":{

"description":"Secretkeytouseforsigningsessiontokens.",

"type":"password"

}

}

Configuration

198

DependencymanagementTherearetwothingscommonlycalled"dependencies"inFoxx:

Moduledependencies,i.e.dependenciesonexternalJavaScriptmodules(e.g.fromthepublicnpmregistry)

Foxxdependencies,i.e.dependenciesbetweenFoxxservices

Let'slookattheminmoredetail:

Moduledependencies

Youcanusethenode_modulesfoldertobundlethird-partyFoxx-compatiblenpmandNode.jsmoduleswithyourFoxxservice.Typicallythisisachievedbyaddingapackage.jsonfiletoyourprojectspecifyingnpmdependenciesusingthedependenciesattributeandinstallingthemwiththenpmcommand-linetool.

Makesuretoincludetheactualnode_modulesfolderinyourFoxxservicebundleasArangoDBwillnotdoanythingspecialtoinstallthesedependencies.AlsokeepinmindthatbundlingextraneousmoduleslikedevelopmentdependenciesmaybloatthefilesizeofyourFoxxservicebundle.

Compatibilitycaveats

UnlikeJavaScriptinbrowsersorNode.js,theJavaScriptenvironmentinArangoDBissynchronous.ThismeansanymodulesthatdependonasynchronousbehaviourlikepromisesorsetTimeoutwillnotbehavecorrectlyinArangoDBorFoxx.AdditionallyunlikeNode.jsArangoDBdoesnotsupportnativeextensions.AllmoduleshavetobeimplementedinpureJavaScript.

WhileArangoDBprovidesalotofcompatibilitycodetosupportmoduleswrittenforNode.js,someNode.jsbuilt-inmodulescannotbeprovidedbyArangoDB.ForacloserlookattheNode.jsmodulesArangoDBdoesordoesnotprovidecheckouttheappendixonJavaScriptmodules.

Alsonotethattheserestrictionsnotonlyapplyonthemodulesyouwishtoinstallbutalsothedependenciesofthosemodules.Asaruleofthumb:moduleswrittentoworkinNode.jsandthebrowserthatdonotrelyonasyncbehaviourshouldgenerallywork;modulesthatrelyonnetworkorfilesystemI/Oormakeheavyuseofasyncbehaviourmostlikelywillnot.

Foxxdependencies

Foxxdependenciescanbedeclaredinaservice'smanifestusingtheprovidesanddependenciesfields:

providesliststhedependenciesagivenserviceprovides,i.e.whichAPIsitclaimstobecompatiblewith

dependenciesliststhedependenciesagivenserviceuses,i.e.whichAPIsitsdependenciesneedtobecompatiblewith

Adependencynameshouldgenerallyusethesameformatasanamespaced(org-scoped)NPMmodule,e.g.@foxx/sessions.

DependencynamesrefertotheexternalJavaScriptAPIofaserviceratherthanspecificservicesimplementingthoseAPIs.Somedependencynamesdefinedbyofficiallymaintainedservicesare:

@foxx/auth(version1.0.0)@foxx/api-keys(version1.0.0)@foxx/bugsnag(versions1.0.0and2.0.0)@foxx/mailgun(versions1.0.0and2.0.0)@foxx/postageapp(versions1.0.0and2.0.0)@foxx/postmark(versions1.0.0and2.0.0)@foxx/sendgrid(versions1.0.0and2.0.0)@foxx/oauth2(versions1.0.0and2.0.0)@foxx/segment-io(versions1.0.0and2.0.0)@foxx/sessions(versions1.0.0and2.0.0)@foxx/users(versions1.0.0,2.0.0and3.0.0)

Dependencies

199

Aprovidesdefinitionmapseachprovideddependency'snametotheprovidedversion:

"provides":{

"@foxx/auth":"1.0.0"

}

Adependenciesdefinitionmapsthelocalaliasofagivendependencyagainstitsnameandthesupportedversionrange(eitherasaJSONobjectorashorthandstring):

"dependencies":{

"mySessions":"@foxx/sessions:^2.0.0",

"myAuth":{

"name":"@foxx/auth",

"version":"^1.0.0",

"description":"Thisdescriptionisentirelyoptional.",

"required":false

}

}

Dependenciescanbeconfiguredfromthewebinterfaceinaservice'ssettingstabusingtheDependenciesbutton.

Thevalueforeachdependencyshouldbethedatabase-relativemountpathoftheservice(includingtheleadingslash).Inordertobeusableasthedependencyofanotherservicebothservicesneedtobemountedinthesamedatabase.Aservicecanbeusedtoprovidemultipledependenciesforthesameservice(aslongastheexpectedJavaScriptAPIsdon'tconflict).

Aservicethathasunconfiguredrequireddependenciescannotbeuseduntilallofitsdependencieshavebeenconfigured.

Itispossibletospecifythemountpathofaservicethatdoesnotactuallydeclarethedependencyasprovided.Thereiscurrentlynovalidationbeyondthemanifestformats.

Whenaserviceusesanothermountedserviceasadependencythedependency'smainentryfile'sexportsobjectbecomesavailableinthemodule.context.dependenciesobjectoftheotherservice:

Examples

ServiceAandServiceBaremountedinthesamedatabase.ServiceBhasadependencywiththelocalalias"greeter".ThedependencyisconfiguredtousethemountpathofServiceA.

//EntryfileofServiceA

module.exports={

sayHi(){

return'Hello';

}

};

//SomewhereinServiceB

constgreeter=module.context.dependencies.greeter;

res.write(greeter.sayHi());

Dependencies

200

RoutersconstcreateRouter=require('@arangodb/foxx/router');

RoutersletyoudefineroutesthatextendArangoDB'sHTTPAPIwithcustomendpoints.

RoutersneedtobemountedusingtheusemethodofaservicecontexttoexposetheirHTTProutesataservice'smountpath.

Youcanpassroutersbetweenservicesmountedinthesamedatabaseasdependencies.Youcanevennestrouterswithineachother.

CreatingaroutercreateRouter():Router

Thisreturnsanew,cleanrouterobjectthathasnotyetbeenmountedintheserviceandcanbeexportedlikeanyotherobject.

Requesthandlersrouter.get([path],[...middleware],handler,[name]):Endpoint

router.post([path],[...middleware],handler,[name]):Endpoint

router.put([path],[...middleware],handler,[name]):Endpoint

router.patch([path],[...middleware],handler,[name]):Endpoint

router.delete([path],[...middleware],handler,[name]):Endpoint

router.all([path],[...middleware],handler,[name]):Endpoint

Thesemethodsletyouspecifyroutesontherouter.TheallmethoddefinesaroutethatwillmatchanysupportedHTTPverb,theothermethodsdefineroutesthatonlymatchtheHTTPverbwiththesamename.

Arguments

path:string(Default:"/")

ThepathoftherequesthandlerrelativetothebasepaththeRouterismountedat.Ifomitted,therequesthandlerwillhandlerequeststothebasepathoftheRouter.Forinformationondefiningdynamicroutesseethesectiononpathparametersinthechapteronrouterendpoints.

middleware:Function(optional)

Zeroormoremiddlewarefunctionsthattakethefollowingarguments:

req:Request

Anincomingserverrequestobject.

res:Response

Anoutgoingserverresponseobject.

next:Function

Acallbackthatpassescontrolovertothenextmiddlewarefunctionandreturnswhenthatfunctionhascompleted.

Ifatruthyargumentispassed,thatargumentwillbethrownasanerror.

Ifthereisnonextmiddlewarefunction,thehandlerwillbeinvokedinstead(seebelow).

handler:Function

Afunctionthattakesthefollowingarguments:

req:Request

Anincomingserverrequestobject.

Routers

201

res:Response

Anoutgoingserverresponse.


AnamethatcanbeusedtogenerateURLsfortheendpoint.Formoreinformationseethereversemethodoftherequestobject.

ReturnsanEndpointfortheroute.

Examples

Simpleindexroute:


res.set('content-type','text/plain');

res.write('HelloWorld!');

});

RestrictingaccesstoauthenticatedArangoDBusers:

router.get('/secrets',function(req,res,next){

if(req.arangoUser){

next();

}else{

res.throw(404,'Secrets?Whatsecrets?');

}

},function(req,res){

res.download('allOurSecrets.zip');

});

Multiplemiddlewarefunctions:

functioncounting(req,res,next){

if(!req.counter)req.counter=0;

req.counter++;

next();

req.counter--;

}

router.get(counting,counting,counting,function(req,res){

res.json({counter:req.counter});//{"counter":3}

});

Mountingchildroutersandmiddlewarerouter.use([path],middleware,[name]):Endpoint

Theusemethodletsyoumountachildrouterormiddlewareatagivenpath.

Arguments

path:string(optional)

ThepathofthemiddlewarerelativetothebasepaththeRouterismountedat.Ifomitted,themiddlewarewillhandlerequeststothebasepathoftheRouter.Forinformationondefiningdynamicroutesseethesectiononpathparametersinthechapteronrouterendpoints.

middleware:Router|Middleware

Anunmountedrouterobjectoramiddleware.


AnamethatcanbeusedtogenerateURLsforendpointsofthisrouter.Formoreinformationseethereversemethodoftherequestobject.HasnoeffectifhandlerisaMiddleware.

ReturnsanEndpointforthemiddlewareorchildrouter.

Routers

202

Routers

203

EndpointsEndpointsarereturnedbytheuse,allandHTTPverb(e.g.get,post)methodsofroutersaswellastheusemethodoftheservicecontext.Theycanbeusedtoattachmetadatatomountedroutes,middlewareandchildroutersthataffectshowrequestsandresponsesareprocessedorprovidesAPIdocumentation.

Endpointsshouldonlybeusedtoinvokethefollowingmethods.Endpointmethodscanbechainedtogether(eachmethodreturnstheendpointitself).

headerendpoint.header(name,[schema],[description]):this

Definesarequestheaderrecognizedbytheendpoint.Anyadditionalnon-definedheaderswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisheaderdefinitionunlessoverridden.

Arguments

name:string

Nameoftheheader.Thisshouldbeconsideredcaseinsensitiveasallheadernameswillbeconvertedtolowercase.

schema:Schema(optional)

Aschemadescribingtheformatoftheheadervalue.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.

Thevalueofthisheaderwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.


AhumanreadablestringthatwillbeshownintheAPIdocumentation.

Returnstheendpoint.

Examples

router.get(/*...*/)

.header('arangoVersion',joi.number().min(30000).default(30000));

pathParamendpoint.pathParam(name,[schema],[description]):this

Definesapathparameterrecognizedbytheendpoint.Pathparametersareexpectedtobefilledaspartoftheendpoint'smountpath.Anyadditionalnon-definedpathparameterswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisparameterdefinitionunlessoverridden.

Arguments

name:string

Nameoftheparameter.


Aschemadescribingtheformatoftheparameter.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.

Endpoints

204

Thevalueofthisparameterwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultintheroutefailingtomatchandbeingignored(resultingina404(NotFound)errorresponseifnootherroutesmatch).



Returnstheendpoint.

Examples

router.get('/some/:num/here',/*...*/)

.pathParam('num',joi.number().required());

queryParamendpoint.queryParam(name,[schema],[description]):this

Definesaqueryparameterrecognizedbytheendpoint.Anyadditionalnon-definedqueryparameterswillbetreatedasoptionalstringvalues.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisparameterdefinitionunlessoverridden.

Arguments

name:string

Nameoftheparameter.


Aschemadescribingtheformatoftheparameter.Thiscanbeajoischemaoranythingthathasacompatiblevalidatemethod.

Thevalueofthisparameterwillbesettothevaluepropertyofthevalidationresult.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.



Returnstheendpoint.

Examples

router.get(/*...*/)

.queryParam('num',joi.number().required());

bodyendpoint.body([model],[mimes],[description]):this

Definestherequestbodyrecognizedbytheendpoint.Therecanonlybeonerequestbodydefinitionperendpoint.ThedefinitionwillalsobeshownintheroutedetailsintheAPIdocumentation.

Intheabsenceofarequestbodydefinition,therequestobject'sbodypropertywillbeinitializedtotheunprocessedrawBodybuffer.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisbodydefinitionunlessoverridden.Iftheendpointisamiddleware,therequestbodywillonlybeparsedonce(i.e.theMIMEtypesoftheroutematchingthesamerequestwillbeignoredbutthebodywillstillbevalidatedagain).

Arguments

model:Model|Schema|null(optional)

Amodelorjoischemadescribingtherequestbody.Avalidationfailurewillresultinanautomatic400(BadRequest)errorresponse.

Endpoints

205

IfthevalueisamodelwithafromClientmethod,thatmethodwillbeappliedtotheparsedrequestbody.

Ifthevalueisaschemaoramodelwithaschema,theschemawillbeusedtovalidatetherequestbodyandthevaluepropertyofthevalidationresultoftheparsedrequestbodywillbeusedinsteadoftheparsedrequestbodyitself.

IfthevalueisamodeloraschemaandtheMIMEtypehasbeenomitted,theMIMEtypewilldefaulttoJSONinstead.

Ifthevalueisexplicitlysettonull,norequestbodywillbeexpected.

Ifthevalueisanarraycontainingexactlyonemodelorschema,therequestbodywillbetreatedasanarrayofitemsmatchingthatmodelorschema.

mimes:Array<string>(optional)

AnarrayofMIMEtypestheroutesupports.

Commonnon-mimealiaseslike"json"or"html"arealsosupportedandwillbeexpandedtotheappropriateMIMEtype(e.g."application/json"and"text/html").

IftheMIMEtypeisrecognizedbyFoxxtherequestbodywillbeparsedintotheappropriatestructurebeforebeingvalidated.CurrentlyonlyJSON,application/x-www-form-urlencodedandmultipartformatsaresupportedinthisway.

IftheMIMEtypeindicatedintherequestheadersdoesnotmatchanyofthesupportedMIMEtypes,thefirstMIMEtypeinthelistwillbeusedinstead.

Failuretoparsetherequestbodywillresultinanautomatic400(BadRequest)errorresponse.



Returnstheendpoint.

Examples

router.post('/expects/some/json',/*...*/)

.body(

joi.object().required(),

'ThisimpliesJSON.'

);

router.post('/expects/nothing',/*...*/)

.body(null);//Nobodyallowed

router.post('/expects/some/plaintext',/*...*/)

.body(['text/plain'],'Thisbodywillbeastring.');

responseendpoint.response([status],[model],[mimes],[description]):this

Definesaresponsebodyfortheendpoint.Whenusingtheresponseobject'ssendmethodintherequesthandlerofthisroute,thedefinitionwiththematchingstatuscodewillbeusedtogeneratetheresponsebody.ThedefinitionswillalsobeshownintheroutedetailsintheAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisresponsedefinitionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.

Arguments

status:number|string(Default:200or204)

HTTPstatuscodetheresponseappliesto.Ifastringisprovidedinsteadofanumericstatuscodeitwillbeusedtolookupanumericstatuscodeusingthestatusesmodule.

model:Model|Schema|null(optional)

Amodelorjoischemadescribingtheresponsebody.

Endpoints

206

https://github.com/jshttp/statuses

IfthevalueisamodelwithaforClientmethod,thatmethodwillbeappliedtothedatapassedtoresponse.sendwithintherouteiftheresponsestatuscodematches(butalsoifnostatuscodehasbeenset).

Ifthevalueisaschemaoramodelwithaschema,theactualschemawillnotbeusedtovalidatetheresponsebodyandonlyservestodocumenttheresponseinmoredetailintheAPIdocumentation.

IfthevalueisamodeloraschemaandtheMIMEtypehasbeenomitted,theMIMEtypewilldefaulttoJSONinstead.

Ifthevalueisexplicitlysettonullandthestatuscodehasbeenomitted,thestatuscodewilldefaultto204("nocontent")insteadof200.

Ifthevalueisanarraycontainingexactlyonemodelorschema,theresponsebodywillbeanarrayofitemsmatchingthatmodelorschema.

mimes:Array<string>(optional)

AnarrayofMIMEtypestheroutemightrespondwithforthisstatuscode.

Commonnon-mimealiaseslike"json"or"html"arealsosupportedandwillbeexpandedtotheappropriateMIMEtype(e.g."application/json"and"text/html").

Whenusingtheresponse.sendmethodtheresponsebodywillbeconvertedtotheappropriateMIMEtypeifpossible.


Ahuman-readablestringthatbrieflydescribestheresponseandwillbeshownintheendpoint'sdetaileddocumentation.

Returnstheendpoint.

Examples

//Thisexampleonlyprovidesdocumentation

//andimpliesagenericJSONresponsebody.

router.get(/*...*/)

.response(

joi.array().items(joi.string()),

'Alistofdoodadidentifiers.'

);

//Noresponsebodywillbeexpectedhere.

router.delete(/*...*/)

.response(null,'Thedoodadnolongerexists.');

//Anendpointcandefinemultipleresponsetypes

//fordifferentstatuscodes--butnevermorethan

//oneforeachstatuscode.

router.post(/*...*/)

.response('found','Thedoodadislocatedelsewhere.')

.response(201,['text/plain'],'Thedoodadwascreatedsohereisahaiku.');

//Heretheresponsebodywillbesetto

//thequerystring-encodedresultof

//FormModel.forClient({some:'data'})

//becausethestatuscodedefaultsto200.

router.patch(function(req,res){

//...

res.send({some:'data'});

})

.response(FormModel,['application/x-www-form-urlencoded'],'OMG.');

//Inthiscasetheresponsebodywillbesetto

//SomeModel.forClient({some:'data'})because

//thestatuscodehasbeensetto201before.

router.put(function(req,res){

//...

res.status(201);

res.send({some:'data'});

})

.response(201,SomeModel,'Somethingamazinghappened.');

error

Endpoints

207

endpoint.error(status,[description]):this

Documentsanerrorstatusfortheendpoint.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethiserrordescriptionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.

ThismethodonlyaffectsthegeneratedAPIdocumentationandhasnotothereffectwithintheserviceitself.

Arguments

status:number|string

HTTPstatuscodefortheerror(e.g.400for"badrequest").Ifastringisprovidedinsteadofanumericstatuscodeitwillbeusedtolookupanumericstatuscodeusingthestatusesmodule.


Ahuman-readablestringthatbrieflydescribestheerrorconditionandwillbeshownintheendpoint'sdetaileddocumentation.

Returnstheendpoint.

Examples


//...

res.throw(403,'Validationerroratx.y.z');

})

.error(403,'Indicatesthatavalidationhasfailed.');

summaryendpoint.summary(summary):this

Addsashortdescriptiontotheendpoint'sAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethissummaryunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.


Arguments

summary:string

Ahuman-readablestringthatbrieflydescribestheendpointandwillappearnexttotheendpoint'spathinthedocumentation.

Returnstheendpoint.

Examples

router.get(/*...*/)

.summary('Listalldiscombobulateddoodads')

descriptionendpoint.description(description):this

Addsalongdescriptiontotheendpoint'sAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillusethisdescriptionunlessoverridden.Iftheendpointisamiddleware,thismethodhasnoeffect.


Arguments

Endpoints

208


description:string

Ahuman-readablestringthatdescribestheendpointindetailandwillbeshownintheendpoint'sdetaileddocumentation.

Returnstheendpoint.

Examples

//The"dedent"libraryhelpsformatting

//multi-linestringsbyadjustingindentation

//andremovingleadingandtrailingblanklines

constdd=require('dedent');

router.post(/*...*/)

.description(dd`

Thisroutediscombobulatesthedoodadsby

frobnicatingthemoxieoftherequestbody.

`)

deprecatedendpoint.deprecated([deprecated]):this

Markstheendpointasdeprecated.

Iftheendpointisachildrouter,allroutesofthatrouterwillalsobemarkedasdeprecated.Iftheendpointisamiddleware,thismethodhasnoeffect.


Arguments

deprecated:boolean(Default:true)

Whethertheendpointshouldbemarkedasdeprecated.Ifsettofalsetheendpointwillbeexplicitlymarkedasnotdeprecated.

Returnstheendpoint.

Examples

router.get(/*...*/)

.deprecated();

tagendpoint.tag(...tags):this

MarkstheendpointwiththegiventagsthatwillbeusedtogrouprelatedroutesinthegeneratedAPIdocumentation.

Iftheendpointisachildrouter,allroutesofthatrouterwillalsobemarkedwiththetags.Iftheendpointisamiddleware,thismethodhasnoeffect.


Arguments

tags:string

Oneormorestringsthatwillbeusedtogrouptheendpoint'sroutes.

Returnstheendpoint.

Examples

router.get(/*...*/)

.tag('auth','restricted');

Endpoints

209

Endpoints

210

MiddlewareMiddlewareinFoxxreferstofunctionsthatcanbemountedlikeroutesandcanmanipulatetherequestandresponseobjectsbeforeandaftertherouteitselfisinvoked.Theycanalsobeusedtocontrolaccessortoprovidecommonlogiclikeloggingetc.Unlikeroutes,middlewareismountedwiththeusemethodlikearouter.

Insteadofafunctiontheusemethodcanalsoacceptanobjectwitharegisterfunctionthatwilltakeaparameterendpoint,themiddlewarewillbemountedatandreturnstheactualmiddlewarefunction.Thisallowsmanipulatingtheendpointbeforecreatingthemiddleware(e.g.todocumentheaders,requestbodies,pathparametersorqueryparameters).

Examples

RestrictaccesstoArangoDB-authenticatedusers:

module.context.use(function(req,res,next){

if(!req.arangoUser){

res.throw(401,'NotauthenticatedwithArangoDB');

}

next();

});

Anytruthyargumentpassedtothenextfunctionwillbethrownasanerror:


leterr=null;


err=newError('Thisshouldneverhappen');

}

next(err);//throwsiftheerrorwasset

})

Trivialloggingmiddleware:


conststart=Date.now();

try{

next();

}finally{

console.log(`Handledrequestin${Date.now()-start}ms`);

}

});

Morecomplexexampleforheader-basedsessions:

constsessions=module.context.collection('sessions');

module.context.use({

register(endpoint){

endpoint.header('x-session-id',joi.string().optional(),'ThesessionID.');

returnfunction(req,res,next){

constsid=req.get('x-session-id');

if(sid){

try{

req.session=sessions.document(sid);

}catch(e){

deletereq.headers['x-session-id'];

}

}

next();

if(req.session){

if(req.session._rev){

sessions.replace(req.session,req.session);

res.set('x-session-id',req.session._key);

}else{

constmeta=sessions.save(req.session);

res.set('x-session-id',meta._key);

}

Middleware

211

}

};

}

});

Middleware

212

RequestobjectsTherequestobjectspecifiesthefollowingproperties:

arangoUser:string|null

TheauthenticatedArangoDBusernameusedtomaketherequest.ThisvalueisonlysetifauthenticationisenabledinArangoDBandtherequestsetanauthorizationheaderArangoDBwasabletoverify.YouarestronglyencouragedtoimplementyourownauthenticationlogicforyourownservicesbutthispropertycanbeusefulifyouneedtointegratewithArangoDB'sownauthenticationmechanisms.

arangoVersion:number

Thenumericvalueofthex-arango-versionheaderorthenumericversionoftheArangoDBserver(e.g.30102forversion3.1.2)ifnovalidheaderwasprovided.

baseUrl:string

Root-relativebaseURLoftheservice,i.e.theprefix"/_db/"followedbythevalueofdatabase.

body:any

Theprocessedandvalidatedrequestbodyforthecurrentroute.Ifnobodyhasbeendefinedforthecurrentroute,thevaluewillbeidenticaltorawBody.

FordetailsonhowrequestbodiescanbeprocessedandvalidatedbyFoxxseethebodymethodoftheendpointobject.

context:Context

Theservicecontextinwhichtherouterwasmounted(ratherthanthecontextinwhichtheroutewasdefined).

database:string

Thenameofthedatabaseinwhichtherequestisbeinghandled,e.g."_system".

headers:object

Therawheadersobject.

FordetailsonhowrequestheaderscanbevalidatedbyFoxxseetheheadermethodoftheendpointobject.

hostname:string

Thehostname(domainname)indicatedintherequestheaders.

Defaultstothehostnameportion(i.e.excludingtheport)oftheHostheaderandfallsbacktothelisteningaddressoftheserver.

method:string

TheHTTPverbusedtomaketherequest,e.g."GET".

originalUrl:string

Root-relativeURLoftherequest,i.e.pathfollowedbytherawqueryparameters,ifany.

path:string

Database-relativepathoftherequestURL(notincludingthequeryparameters).

pathParams:object

Anobjectmappingthenamesofpathparametersofthecurrentroutetotheirvalidatedvalues.

FordetailsonhowpathparameterscanbevalidatedbyFoxxseethepathParammethodoftheendpointobject.

port:number

Theportindicatedintherequestheaders.

Request

213

Defaultstotheportportion(i.e.excludingthehostname)oftheHostheaderandfallsbacktothelisteningportortheappropriatedefaultport(443forHTTPSor80forHTTP,dependingonsecure)iftheheaderonlyindicatesahostname.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettotheportportionoftheX-Forwarded-Hostheader(orapproriatedefaultport)ifpresent.

protocol:string

Theprotocolusedfortherequest.

Defaultsto"https"or"http"dependingonwhetherArangoDBisconfiguredtouseSSLornot.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothevalueoftheX-Forwarded-Protoheaderifpresent.

queryParams:object

Anobjectmappingthenamesofqueryparametersofthecurrentroutetotheirvalidatedvalues.

FordetailsonhowqueryparameterscanbevalidatedbyFoxxseethequeryParammethodoftheendpointobject.

rawBody:Buffer

Theraw,unparsed,unvalidatedrequestbodyasabuffer.

remoteAddress:string

TheIPoftheclientthatmadetherequest.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothefirstIPlistedintheX-Forwarded-Forheaderifpresent.

remoteAddresses:Array<string>

AlistcontainingtheIPaddressesusedtomaketherequest.

DefaultstothevalueofremoteAddresswrappedinanarray.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettothelistofIPsspecifiedintheX-Forwarded-Forheaderifpresent.

remotePort:number

Thelisteningportoftheclientthatmadetherequest.

Iftherequestwasmadeusingatrustedproxy(seetrustProxy),thisissettotheportspecifiedintheX-Forwarded-Portheaderifpresent.

secure:boolean

Whethertherequestwasmadeoverasecureconnection(i.e.HTTPS).

Thisissettofalsewhenprotocolis"http"andtruewhenprotocolis"https".

suffix:string

Thetrailingpathrelativetothecurrentrouteifthecurrentrouteendsinawildcard(e.g./something/*).

Note:StartingwithArangoDB3.2ispassedintotheserviceas-is,i.e.percentageescapesequenceslike%2Fwillnolongerbeunescaped.Alsonotethatthesuffixmaycontainpathsegmentslike..whichmayhavespecialmeaningifthesuffixisusedtobuildfilesystempaths.

trustProxy:boolean

Indicateswhethertherequestwasmadeusingatrustedproxy.Iftheoriginserver'saddresswasspecifiedintheArangoDBconfigurationusing--frontend.trusted-proxyortheservice'strustProxysettingisenabled,thiswillbetrue,otherwiseitwillbefalse.

url:string

TheURLoftherequest.

Request

214

xhr:boolean

WhethertherequestindicatesitwasmadewithinabrowserusingAJAX.

ThisissettotrueiftheX-Requested-Withheaderispresentandisacase-insensitivematchforthevalue"xmlhttprequest".

NotethatthisvaluedoesnotguaranteewhethertherequestwasmadefrominsideabrowserorwhetherAJAXwasusedandismerelyaconventionestablishedbyJavaScriptframeworkslikejQuery.

acceptsreq.accepts(types):string|false

req.accepts(...types):string|false

req.acceptsCharsets(charsets):string|false

req.acceptsCharsets(...charsets):string|false

req.acceptsEncodings(encodings):string|false

req.acceptsEncodings(...encodings):string|false

req.acceptsLanguages(languages):string|false

req.acceptsLanguages(...languages):string|false

Thesemethodswrapthecorrespondingcontentnegotiationmethodsoftheacceptsmoduleforthecurrentrequest.

Examples

if(req.accepts(['json','html'])==='html'){

//ClientexplicitlyprefersHTMLoverJSON

res.write('<h1>ClientprefersHTML</h1>');

}else{

//OtherwisejustsendJSON

res.json({success:true});

}

cookiereq.cookie(name,options):string|null

Getsthevalueofacookiebyname.

Arguments

name:string

Nameofthecookie.

options:object(optional)


secret:string(optional)

Secretthatwasusedtosignthecookie.

Ifasecretisspecified,thecookie'ssignatureisexpectedtobepresentinasecondcookiewiththesamenameandthesuffix.sig.Otherwisethesignature(ifpresent)willbeignored.

algorithm:string(Default:"sha256")

Algorithmthatwasusedtosignthecookie.

Ifastringispassedinsteadofanoptionsobjectitwillbeinterpretedasthesecretoption.

Returnsthevalueofthecookieornullifthecookieisnotsetoritssignatureisinvalid.

Request

215

https://github.com/jshttp/accepts

get/headerreq.get(name):string

req.header(name):string

Getsthevalueofaheaderbyname.Youcanvalidaterequestheadersusingtheheadermethodoftheendpoint.

Arguments

name:string

Nameoftheheader.

Returnstheheadervalue.

isreq.is(types):string

req.is(...types):string

Thismethodwrapsthe(requestbody)contenttypedetectionmethodofthetype-ismoduleforthecurrentrequest.

Examples

consttype=req.is('html','application/xml','application/*+xml');

if(type===false){//nomatch

handleDefault(req.rawBody);

}elseif(type==='html'){

handleHtml(req.rawBody);

}else{//isXML

handleXml(req.rawBody);

}

jsonreq.json():any

AttemptstoparsetherawrequestbodyasJSONandreturnstheresult.

Itisgenerallymoreusefultodefinearequestbodyontheendpointandusethereq.bodypropertyinstead.

Returnsundefinediftherequestbodyisempty.MaythrowaSyntaxErrorifthebodycouldnotbeparsed.

makeAbsolutereq.makeAbsolute(path,[query]):string

Resolvesthegivenpathrelativetothereq.context.service'smountpathtoafullURL.

Arguments

path:string

Thepathtoresovle.

query:string|object

AstringorobjectwithqueryparameterstoaddtotheURL.

ReturnstheformattedabsoluteURL.

paramsreq.param(name):any

Request

216

https://github.com/jshttp/type-is

Arguments

Looksupaparameterbyname,preferringpathParamsoverqueryParams.

It'sprobablybetterstyletousethereq.pathParamsorreq.queryParamsobjectsdirectly.

name:string

Nameoftheparameter.

Returnsthe(validated)valueoftheparameter.

rangereq.range([size]):Ranges|number

Thismethodwrapstherangeheaderparsingmethodoftherange-parsermoduleforthecurrentrequest.

Arguments

size:number(Default:Infinity)

Lengthofthesatisfiablerange(e.g.numberofbytesinthefullresponse).Ifpresent,rangesexceedingthesizewillbeconsideredunsatisfiable.

ReturnsundefinediftheRangeheaderisabsent,-2iftheheaderispresentbutmalformed,-1iftherangeisinvalid(e.g.startoffsetislargerthanendoffset)orunsatisfiableforthegivensize.

Otherwisereturnsanarrayofobjectswiththepropertiesstartandendvaluesforeachrange.Thearrayhasanadditionalpropertytypeindicatingtherequestrangetype.

Examples

console.log(req.headers.range);//"bytes=40-80"

constranges=req.range(100);

console.log(ranges);//[{start:40,end:80}]

console.log(ranges.type);//"bytes"

reversereq.reverse(name,[params]):string

LooksuptheURLofanamedrouteforthegivenparameters.

Arguments

name:string

Nameoftheroutetolookup.

params:object(optional)

Anobjectcontainingvaluesforthe(pathorquery)parametersoftheroute.

ReturnstheURLoftherouteforthegivenparameters.

Examples

router.get('/items/:id',function(req,res){

/*...*/

},'getItemById');

router.post('/items',function(req,res){

//...

consturl=req.reverse('getItemById',{id:createdItem._key});

res.set('location',req.makeAbsolute(url));

});

Request

217

https://github.com/jshttp/range-parser

Request

218

ResponseobjectsTheresponseobjectspecifiesthefollowingproperties:

body:Buffer|string

Responsebodyasastringorbuffer.Canbesetdirectlyorusingsomeoftheresponsemethods.

context:Context

Theservicecontextinwhichtherouterwasmounted(ratherthanthecontextinwhichtheroutewasdefined).

headers:object

Therawheadersobject.

statusCode:number

Statuscodeoftheresponse.Defaultsto200(bodysetandnotanemptystringorbuffer)or204(otherwise)ifnotchangedfromundefined.

attachmentres.attachment([filename]):this

Setsthecontent-dispositionheadertoindicatetheresponseisadownloadablefilewiththegivenname.

Note:Thisdoesnotactuallymodifytheresponsebodyoraccessthefilesystem.TosendafilefromthefilesystemseethedownloadorsendFilemethods.

Arguments

filename:string(optional)

Nameofthedownloadablefileintheresponsebody.

Ifpresent,theextensionofthefilenamewillbeusedtosettheresponsecontent-typeifithasnotyetbeenset.

Returnstheresponseobject.

cookieres.cookie(name,value,[options]):this

Setsacookiewiththegivenname.

Arguments

name:string

Nameofthecookie.

value:string

Valueofthecookie.



ttl:number(optional)

Timetoliveofthecookieinseconds.

algorithm:string(Default:"sha256")

Algorithmthatwillbeusedtosignthecookie.

Response

219


Secretthatwillbeusedtosignthecookie.

Ifasecretisspecified,thecookie'ssignaturewillbestoredinasecondcookiewiththesameoptions,thesamenameandthesuffix.sig.Otherwisenosignaturewillbeadded.


Pathforwhichthecookieshouldbeissued.

domain:string(optional)

Domainforwhichthecookieshouldbeissued.

secure:boolean(Default:false)

Whetherthecookieshouldbemarkedassecure(i.e.HTTPS/SSL-only).

httpOnly:boolean(Default:false)

WhetherthecookieshouldbemarkedasHTTP-only(ratherthanalsoexposingittoclient-sidecode).


Ifanumberispassedinsteadofanoptionsobjectitwillbeinterpretedasthettloption.


downloadres.download(path,[filename]):this

Theequivalentofcallingres.attachment(filename).sendFile(path).

Arguments

path:string

Pathtothefileonthelocalfilesystemtobesentastheresponsebody.

filename:string(optional)

Filenametoindicateinthecontent-dispositionheader.

Ifomittedthepathwillbeusedinstead.


getHeaderres.getHeader(name):string

Getsthevalueoftheheaderwiththegivenname.

Arguments

name:string

Nameoftheheadertoget.

Returnsthevalueoftheheaderorundefined.

jsonres.json(data):this

SetstheresponsebodytotheJSONstringvalueofthegivendata.

Response

220

Arguments

data:any

Thedatatobeusedastheresponsebody.


redirectres.redirect([status],path):this

Redirectstheresponsebysettingtheresponselocationheaderandstatuscode.

Arguments

status:number|string(optional)

Responsestatuscodetoset.

Ifthestatuscodeisthestringvalue"permanent"itwillbetreatedasthevalue301.

Ifthestatuscodeisastringitwillbeconvertedtoanumericstatuscodeusingthestatusesmodulefirst.

Ifthestatuscodeisomittedbuttheresponsestatushasnotalreadybeenset,theresponsestatuswillbesetto302.

path:string

URLtosetthelocationheaderto.


removeHeaderres.removeHeader(name):this

Removestheheaderwiththegivennamefromtheresponse.

Arguments

name:string

Nameoftheheadertoremove.


sendres.send(data,[type]):this

Setstheresponsebodytothegivendatawithrespecttotheresponsedefinitionfortheresponse'scurrentstatuscode.

Arguments

data:any

Thedatatobeusedastheresponsebody.Willbeconvertedaccordingtheresponsedefinitionfortheresponse'scurrentstatuscode(or200)inthefollowingway:

IfthedataisanArangoDBresultset,itwillbeconvertedtoanarrayfirst.

IftheresponsedefinitionspecifiesamodelwithaforClientmethod,thatmethodwillbeappliedtothedatafirst.Ifthedataisanarrayandtheresponsedefinitionhasthemultipleflagset,themethodwillbeappliedtoeachentryindividuallyinstead.

Finallythedatawillbeprocessedbytheresponsetypehandlertoconvertheresponsebodytoastringorbuffer.

type:string(Default:"auto")

Response

221


Content-typeoftheresponsebody.

Ifsetto"auto"thefirstMIMEtypespecifiedintheresponsedefinitionfortheresponse'scurrentstatuscode(or200)willbeusedinstead.

Ifsetto"auto"andnoresponsedefinitionexists,theMIMEtypewillbedeterminedthefollowingway:

IfthedataisabuffertheMIMEtypewillbesettobinary(application/octet-stream).

IfthedataisanobjecttheMIMEtypewillbesettoJSONandthedatawillbeconvertedtoaJSONstring.

OtherwisetheMIMEtypewillbesettoHTMLandthedatawillbeconvertedtoastring.


sendFileres.sendFile(path,[options]):this

Sendsafilefromthelocalfilesystemastheresponsebody.

Arguments

path:string

Pathtothefileonthelocalfilesystemtobesentastheresponsebody.

Ifnocontent-typeheaderhasbeensetyet,theextensionofthefilenamewillbeusedtosetthevalueofthatheader.



lastModified:boolean(optional)

Ifsettotrueorifnolast-modifiedheaderhasbeensetyetandthevalueisnotsettofalsethelast-modifiedheaderwillbesettothemodificationdateofthefileinmilliseconds.


Examples

//Sendthefile"favicon.ico"fromthisservice'sfolder

res.sendFile(module.context.fileName('favicon.ico'));

sendStatusres.sendStatus(status):this

Sendsaplaintextresponseforthegivenstatuscode.Theresponsestatuswillbesettothegivenstatuscode,theresponsebodywillbesettothestatusmessagecorrespondingtothatstatuscode.

Arguments





setHeader/setres.setHeader(name,value):this

res.set(name,value):this

Response

222


res.set(headers):this

Setsthevalueoftheheaderwiththegivenname.

Arguments

name:string

Nameoftheheadertoset.

value:string

Valuetosettheheaderto.

headers:object

Headerobjectmappingheadernamestovalues.


statusres.status(status):this

Setstheresponsestatustothegivenstatuscode.

Arguments





throwres.throw(status,[reason],[options]):void

ThrowsanHTTPexceptionforthegivenstatus,whichwillbehandledbyFoxxtoservetheappropriateJSONerrorresponse.

Arguments




Ifthestatuscodeisinthe500-range(500-599),itsstacktracewillalwaysbeloggedasifitwereanunhandledexception.

Ifdevelopmentmodeisenabled,theerror'sstacktracewillbeloggedasawarningifthestatuscodeisinthe400-range(400-499)orasaregularmessageotherwise.

reason:string(optional)

Messagefortheexception.

Ifomitted,thestatusmessagecorrespondingtothestatuscodewillbeusedinstead.



cause:Error(optional)

Causeoftheexceptionthatwillbeloggedaspartoftheerror'sstacktrace(recursively,iftheexceptionalsohasacausepropertyandsoon).

Response

223



extra:object(optional)

AdditionalpropertiesthatwillbeaddedtotheerrorresponsebodygeneratedbyFoxx.

Ifdevelopmentmodeisenabled,anexceptionpropertywillbeaddedtothisvaluecontainingtheerrormessageandastacktracepropertywillbeaddedcontaininganarraywitheachlineoftheerror'sstacktrace.

Ifanerrorispassedinsteadofanoptionsobjectitwillbeinterpretedasthecauseoption.Ifnoreasonwasprovidedtheerror'smessagewillbeusedasthereasoninstead.

Returnsnothing.

typeres.type([type]):string

Setstheresponsecontent-typetothegiventypeifprovidedorreturnsthepreviouslysetcontent-type.

Arguments

type:string(optional)

Content-typeoftheresponsebody.

Unlikeres.set('content-type',type)fileextensionscanbeprovidedasvaluesandwillbetranslatedtothecorrespondingMIMEtype(e.g.jsonbecomesapplication/json).

Returnsthecontent-typeoftheresponsebody.

varyres.vary(names):this

res.vary(...names):this

Thismethodwrapsthevaryheadermanipulationmethodofthevarymoduleforthecurrentresponse.

Thegivennameswillbeaddedtotheresponse'svaryheaderifnotalreadypresent.


Examples

res.vary('user-agent');

res.vary('cookie');

res.vary('cookie');//duplicateswillbeignored

//--or--

res.vary('user-agent','cookie');

//--or--

res.vary(['user-agent','cookie']);

writeres.write(data):this

Appendsthegivendatatotheresponsebody.

Arguments

data:string|Buffer

Datatoappend.

Response

224

https://github.com/jshttp/vary

Ifthedataisabuffertheresponsebodywillbeconvertedtoabufferfirst.

Iftheresponsebodyisabufferthedatawillbeconvertedtoabufferfirst.

IfthedataisanobjectitwillbeconvertedtoaJSONstringfirst.

Ifthedataisanyothernon-stringvalueitwillbeconvertedtoastringfirst.


Response

225

UsingGraphQLinFoxxconstcreateGraphQLRouter=require('@arangodb/foxx/graphql');

Foxxbundlesversion0.6ofthegraphql-syncmodule,whichisasynchronouswrapperfortheofficialJavaScriptGraphQLreferenceimplementation,toallowwritingGraphQLschemasdirectlyinsideFoxx.

Additionallythe@arangodb/foxx/graphqlletsyoucreateroutersforservingGraphQLrequests,whichcloselymimicksthebehaviouroftheexpress-graphqlmodule.

Formoreinformationongraphql-syncseethegraphql-jsAPIreference(notethatgraphql-syncneverwrapsresultsinpromises).

ForanexampleofaGraphQLschemainFoxxthatresolvesfieldsusingthedatabaseseetheGraphQLexampleservice(alsoavailablefromtheFoxxstore).

Examples

constgraphql=require('graphql-sync');

constgraphqlSchema=newgraphql.GraphQLSchema({

//...

});

//Mountingagraphqlendpointdirectlyinaservice:

module.context.use('/graphql',createGraphQLRouter({

schema:graphqlSchema,

graphiql:true

}));

//Orattheservice'srootURL:

module.context.use(createGraphQLRouter({


graphiql:true

}));

//Orinsideanexistingrouter:

router.get('/hello',function(req,res){

res.write('Helloworld!');

});

router.use('/graphql',createGraphQLRouter({


graphiql:true

}));

Note:ArangoDBaimsforstabilitywhichmeansbundleddependencieswillgenerallynotbeupdatedasquicklyastheirmaintainersmakeupdatesavailableonGitHuborNPM.StartingwithArangoDB3.2,ifyouwanttouseanewerversionthantheonebundledwithyourtargetversionofArangoDB,youcanprovideyourownversionofthelibrarybypassingitviathegraphqloption:

constgraphql=require('graphql-sync');

constgraphqlSchema=newgraphql.Schema({

//...

});



graphiql:true,

graphql:graphql

}))

Startingwithgraphql0.12youcanalsousetheofficialgraphqllibraryifyouincludeitinthenode_modulesfolderofyourservicebundle:

constgraphql=require('graphql');//0.12orlater

constgraphqlSchema=newgraphql.Schema({

//...

});



graphiql:true,

UsingGraphQL

226

https://github.com/arangodb/graphql-sync

https://github.com/graphql/express-graphql

http://graphql.org/docs/api-reference-graphql/

https://github.com/arangodb-foxx/demo-graphql

https://github.com/graphql/graphql-js

graphql:graphql

}))

CreatingaroutercreateGraphQLRouter(options):Router

ThisreturnsanewrouterobjectwithPOSTandGETroutesforservingGraphQLrequests.

Arguments

options:object


schema:GraphQLSchema

AGraphQLSchemaobjectfromgraphql-sync.

context:any(optional)

TheGraphQLcontextthatwillbepassedtothegraphql()functionfromgraphql-synctohandleGraphQLqueries.

rootValue:object(optional)

TheGraphQLrootvaluethatwillbepassedtothegraphql()functionfromgraphql-synctohandleGraphQLqueries.

pretty:boolean(Default:false)

Iftrue,JSONresponseswillbepretty-printed.

formatError:Function(optional)

Afunctionthatwillbeusedtoformaterrorsproducedbygraphql-sync.Ifomitted,theformatErrorfunctionfromgraphql-syncwillbeusedinstead.

validationRules:Array<any>(optional)

AdditionalvalidationrulesqueriesmustsatisfyinadditiontothosedefinedintheGraphQLspec.

graphiql:boolean(Default:false)

Iftrue,theGraphiQLexplorerwillbeservedwhenloadeddirectlyfromabrowser.

graphql:object(optional)

Ifyouneedtouseyourowncopyofthegraphql-syncmoduleinsteadoftheonebundledwithArangoDB,hereyoucanpassitindirectly.

IfaGraphQLSchemaobjectispassedinsteadofanoptionsobjectitwillbeinterpretedastheschemaoption.

Generatedroutes

TherouterhandlesGETandPOSTrequeststoitsrootpathandacceptsthefollowingparameters,whichcanbeprovidedeitherasqueryparametersorasthePOSTrequestbody:

query:string

AGraphQLquerythatwillbeexecuted.

variables:object|string(optional)

AnobjectorastringcontainingaJSONobjectwithruntimevaluestouseforanyGraphQLqueryvariables.

operationName:string(optional)

Iftheprovidedquerycontainsmultiplenamedoperations,thisspecifieswhichoperationshouldbeexecuted.

raw:boolean(Default:false)

UsingGraphQL

227

https://github.com/graphql/graphiql

ForcesaJSONresponseevenifgraphiqlisenabledandtherequestwasmadeusingabrowser.

ThePOSTrequestbodycanbeprovidedasJSONorasquerystringusingapplication/x-www-form-urlencoded.Arequestbodypassedasapplication/graphqlwillbeinterpretedasthequeryparameter.

UsingGraphQL

228

SessionMiddlewareconstsessionMiddleware=require('@arangodb/foxx/sessions');

ThesessionmiddlewareaddsthesessionandsessionStoragepropertiestotherequestobjectanddealswithserializinganddeserializingthesessionaswellasextractingsessionidentifiersfromincomingrequestsandinjectingthemintooutgoingresponses.

Examples

//Createasessionmiddleware

constsessions=sessionsMiddleware({

storage:module.context.collection('sessions'),

transport:['header','cookie']

});

//Firstenablethemiddlewareforthisservice

module.context.use(sessions);

//Nowmounttheroutersthatusethesession



router.get('/',function(req,res){

res.send(`Hello${req.session.uid||'anonymous'}!`);

},'hello');

router.post('/login',function(req,res){

req.session.uid=req.body;

req.sessionStorage.save(req.session);

res.redirect(req.reverse('hello'));

});

.body(['text/plain'],'Username');

CreatingasessionmiddlewaresessionMiddleware(options):Middleware

Createsasessionmiddleware.

Arguments

options:Object

Anobjectwiththefollowingproperties:

storage:Storage

Storagethatwillbeusedtopersistthesessions.

ThestorageisalsoexposedasthesessionStorageonallrequestobjectsandasthestoragepropertyofthemiddleware.

IfastringorcollectionispassedinsteadofaStorage,itwillbeusedtocreateaCollectionStorage.

transport:Transport|Array<Transport>

Transportorarrayoftransportsthatwillbeusedtoextractthesessionidentifiersfromincomingrequestsandinjectthemintooutgoingresponses.Whenattemptingtoextractasessionidentifier,thetransportswillbeusedintheorderspecifieduntilamatchisfound.Wheninjecting(orclearing)sessionidentifiers,alltransportswillbeinvoked.

Thetransportsarealsoexposedasthetransportpropertyofthemiddleware.

Ifthestring"cookie"ispassedinsteadofaTransport,theCookieTransportwillbeusedwiththedefaultsettingsinstead.

Ifthestring"header"ispassedinsteadofaTransport,theHeaderTransportwillbeusedwiththedefaultsettingsinstead.

autoCreate:boolean(Default:true)

Ifenabledthesessionstorage'snewmethodwillbeinvokedtocreateanemptysessionwheneverthetransportfailedtoreturnasessionfortheincomingrequest.Otherwisethesessionwillbeinitializedasnull.

Sessionsmiddleware

229

Returnsthesessionmiddleware.

Sessionsmiddleware

230

SessionStoragesSessionstoragesareusedbythesessionsmiddlewaretopersistsessionsacrossrequests.SessionstoragesmustimplementthefromClientandforClientmethodsandcanoptionallyimplementthenewmethod.

Thebuilt-insessionstoragesgenerallyprovidethefollowingattributes:

uid:string(Default:null)

Auniqueidentifierindicatingtheactiveuser.

created:number(Default:Date.now())

Thenumerictimestampofwhenthesessionwascreated.

data:any(Default:null)

Arbitrarydatatopersistedinthesession.

newstorage.new():Session

Generatesanewsessionobjectrepresentinganemptysession.Theemptysessionobjectshouldnotbepersistedunlessnecessary.Thereturnvaluewillbeexposedbythemiddlewareasthesessionpropertyoftherequestobjectifnosessionidentifierwasreturnedbythesessiontransportsandauto-creationisnotexplicitlydisabledinthesessionmiddleware.

Examples

new(){

return{

uid:null,

created:Date.now(),

data:null

};

}

fromClientstorage.fromClient(sid):Session|null

Resolvesordeserializesasessionidentifiertoasessionobject.

Arguments

sid:string

Sessionidentifiertoresolveordeserialize.

Returnsasessionobjectrepresentingthesessionwiththegivensessionidentifierthatwillbeexposedbythemiddlewareasthesessionpropertyoftherequestobject.Thismethodwillonlybecalledifanyofthesessiontransportsreturnedasessionidentifier.Ifthesessionidentifierisinvalidorexpired,themethodshouldreturnanullvaluetoindicatenomatchingsession.

Examples

fromClient(sid){

returndb._collection('sessions').firstExample({_key:sid});

}

forClientstorage.forClient(session):string|null

Sessionstorages

231

Derivesasessionidentifierfromthegivensessionobject.

Arguments

session:Session

Sessiontoderiveasessionidentifierfrom.

Returnsasessionidentifierforthesessionrepresentedbythegivensessionobject.Thismethodwillbecalledwiththesessionpropertyoftherequestobjectunlessthatpropertyisempty(e.g.null).

Examples

forClient(session){

if(!session._key){

constmeta=db._collection('sessions').save(session);

returnmeta._key;

}

db._collection('sessions').replace(session._key,session);

returnsession._key;

}

Sessionstorages

232

CollectionSessionStorageconstcollectionStorage=require('@arangodb/foxx/sessions/storages/collection');

Thecollectionsessionstoragepersistssessionstoacollectioninthedatabase.

CreatingastoragecollectionStorage(options):Storage

CreatesaStoragethatcanbeusedinthesessionsmiddleware.

Arguments

options:Object


collection:ArangoCollection

Thecollectionthatshouldbeusedtopersistthesessions.Ifastringispassedinsteadofacollectionitisassumedtobethefullyqualifiednameofacollectioninthecurrentdatabase.

ttl:number(Default:60*60)

Thetimeinsecondssincethelastupdateuntilasessionwillbeconsideredexpired.

pruneExpired:boolean(Default:false)

Whetherexpiredsessionsshouldberemovedfromthecollectionwhentheyareaccessedinsteadofsimplybeingignored.

autoUpdate:boolean(Default:true)

Whethersessionsshouldbeupdatedinthecollectioneverytimetheyareaccessedtokeepthemfromexpiring.Disablingthisoptionwillimproveperformancebutmeansyouwillhavetotakecareofkeepingyoursessionsaliveyourself.

Ifastringorcollectionispassedinsteadofanoptionsobject,itwillbeinterpretedasthecollectionoption.

prunestorage.prune():Array<string>

Removesallexpiredsessionsfromthecollection.ThismethodshouldbecalledevenifthepruneExpiredoptionisenabledtocleanupabandonedsessions.

Returnsanarrayofthekeysofallsessionsthatwereremoved.

savestorage.save(session):Session

Saves(replaces)thegivensessionobjectinthecollection.Thismethodneedstobeinvokedexplicitlyaftermakingchangestothesessionorthechangeswillnotbepersisted.Assignsanew_keytothesessionifitpreviouslydidnothaveone.

Arguments

session:Session

Asessionobject.

Returnsthemodifiedsession.

clear

Sessionstorages

233

storage.clear(session):boolean

Removesthesessionfromthecollection.Hasnoeffectifthesessionwasalreadyremovedorhasnotyetbeensavedtothecollection(i.e.hasno_key).

Arguments

session:Session

Asessionobject.

Returnstrueifthesessionwasremovedorfalseifithadnoeffect.

Sessionstorages

234

JWTSessionStorageconstjwtStorage=require('@arangodb/foxx/sessions/storages/jwt');

TheJWTsessionstorageconvertssessionstoandfromJSONWebTokens.

Examples

//PassinasecuresecretfromtheFoxxconfiguration

constsecret=module.context.configuration.jwtSecret;


storage:jwtStorage(secret),

transport:'header'

});


CreatingastoragejwtStorage(options):Storage

CreatesaStoragethatcanbeusedinthesessionsmiddleware.

Note:whilethe"none"algorithm(i.e.nosignature)issupportedthisdummyalgorithmprovidesnosecurityandallowsclientstomakearbitrarymodificationstothepayloadandshouldnotbeusedunlessyouarecertainyouspecificallyneedit.

Arguments

options:Object


algorithm:string(Default:"HS512")

Thealgorithmtouseforsigningthetoken.

Supportedvalues:

"HS256"(HMAC-SHA256)"HS384"(HMAC-SHA384)"HS512"(HMAC-SHA512)"none"(nosignature)

secret:string

Thesecrettouseforsigningthetoken.

Thisfieldisforbiddenwhenusingthe"none"algorithmbutrequiredotherwise.

ttl:number(Default:3600)

Themaximumlifetimeofthetokeninseconds.Youmaywanttokeepthisshortasanewtokenisgeneratedoneveryrequestallowingclientstorefreshtokensautomatically.

verify:boolean(Default:true)

Ifsettofalsethesignaturewillnotbeverifiedbutstillgenerated(unlessusingthe"none"algorithm).

maxExp:number(Default:Infinity)

LargestvaluethatwillbeacceptedinanincomingJWTexp(expiration)field.


Sessionstorages

235

https://jwt.io/

SessionTransportsSessiontransportsareusedbythesessionsmiddlewaretostoreandretrievesessionidentifiersinrequestsandresponses.Sessiontransportsmustimplementthegetand/orsetmethodsandcanoptionallyimplementtheclearmethod.

gettransport.get(request):string|null

Retrievesasessionidentifierfromarequestobject.

Ifpresentthismethodwillautomaticallybeinvokedforeachtransportuntilatransportreturnsasessionidentifier.

Arguments

request:Request

Requestobjecttoextractasessionidentifierfrom.

Returnsthesessionidentifierornullifthetransportcannotfindasessionidentifierintherequest.

Examples

get(req){

returnreq.get('x-session-id')||null;

}

settransport.set(response,sid):void

Attachesasessionidentifiertoaresponseobject.

Ifpresentthismethodwillautomaticallybeinvokedattheendofarequestregardlessofwhetherthesessionwasmodifiedornot.

Arguments

response:Response

Responseobjecttoattachasessionidentifierto.

sid:string

Sessionidentifiertoattachtotheresponse.

Returnsnothing.

Examples

set(res){

res.set('x-session-id',value);

}

cleartransport.clear(response):void

Attachesapayloadindicatingthatthesessionhasbeenclearedtotheresponseobject.Thiscanbeusedtoclearasessioncookiewhenthesessionhasbeendestroyed(e.g.duringlogout).

Ifpresentthismethodwillautomaticallybeinvokedinsteadofsetwhenthereq.sessionattributewasremovedbytheroutehandler.

Arguments

Sessiontransports

236

response:Response

Responseobjecttoremovethesessionidentifierfrom.

Returnsnothing.

Sessiontransports

237

CookieSessionTransportconstcookieTransport=require('@arangodb/foxx/sessions/transports/cookie');

Thecookietransportstoressessionidentifiersincookiesontherequestandresponseobject.

Examples

//PassinasecuresecretfromtheFoxxconfiguration

constsecret=module.context.configuration.cookieSecret;



transport:cookieTransport({

name:'FOXXSESSID',

ttl:60*60*24*7,//oneweekinseconds

algorithm:'sha256',

secret:secret

})

});


CreatingatransportcookieTransport([options]):Transport

CreatesaTransportthatcanbeusedinthesessionsmiddleware.

Arguments



name:string(Default:"sid")

Thenameofthecookie.

ttl:number(optional)

Cookielifetimeinseconds.NotethatthisdoesnotaffectthestorageTTL(i.e.howlongthesessionitselfisconsideredvalid),justhowlongthecookieshouldbestoredbytheclient.

algorithm:string(optional)

Thealgorithmusedtosignandverifythecookie.Ifnoalgorithmisspecified,thecookiewillnotbesignedorverified.Seethecookiemethodontheresponseobject.


Secrettouseforthesignedcookie.Willbeignoredifnoalgorithmisprovided.


Pathforwhichthecookieshouldbeissued.

domain:string(optional)

Domainforwhichthecookieshouldbeissued.

secure:boolean(Default:false)

Whetherthecookieshouldbemarkedassecure(i.e.HTTPS/SSL-only).

httpOnly:boolean(Default:false)

WhetherthecookieshouldbemarkedasHTTP-only(ratherthanalsoexposingittoclient-sidecode).

Ifastringispassedinsteadofanoptionsobject,itwillbeinterpretedasthenameoption.

Sessiontransports

238

Sessiontransports

239

HeaderSessionTransportconstheaderTransport=require('@arangodb/foxx/sessions/transports/header');

Theheadertransportstoressessionidentifiersinheadersontherequestandresponseobjects.

Examples



transport:headerTransport('X-FOXXSESSID')

});


CreatingatransportheaderTransport([options]):Transport

CreatesaTransportthatcanbeusedinthesessionsmiddleware.

Arguments



name:string(Default:X-Session-Id)

Nameoftheheaderthatcontainsthesessionidentifier(notcasesensitive).

Ifastringispassedinsteadofanoptionsobject,itwillbeinterpretedasthenameoption.

Sessiontransports

240

StaticfileassetsThemostflexiblewaytoservefilesinyourFoxxserviceistosimplypassthemthroughinyourrouterusingthecontextobject'sfileNamemethodandtheresponseobject'ssendFilemethod:

router.get('/some/filename.png',function(req,res){

constfilePath=module.context.fileName('some-local-filename.png');

res.sendFile(filePath);

});

Whileallowingforgreatercontrolofhowthefileshouldbesenttotheclientandwhoshouldbeabletoaccessit,doingthisforallyourstaticassetscangettedious.

AlternativelyyoucanspecifyfileassetsthatshouldbeservedbyyourFoxxservicedirectlyintheservicemanifestusingthefilesattribute:

"files":{

"/some/filename.png":{

"path":"some-local-filename.png",

"type":"image/png",

"gzip":false

},

"/favicon.ico":"bookmark.ico",

"/static":"my-assets-folder"

}

Eachentryinthefilesattributecanrepresenteitherasinglefileoradirectory.Whenservingentiredirectories,thekeyactsasaprefixandrequeststothatprefixwillberesolvedwithinthegivendirectory.

Options

path:string

Therelativepathofthefileorfolderwithintheservice.

type:string(optional)

TheMIMEcontenttypeofthefile.Defaultstoanintelligentguessbasedonthefilename'sextension.

gzip:boolean(Default:false)

Ifsettotruethefilewillbeservedwithgzip-encodingifsupportedbytheclient.Thiscanbeusefulwhenservingtextfileslikeclient-sideJavaScript,CSSorHTML.

Ifastringisprovidedinsteadofanobject,itwillbeinterpretedasthepathoption.

Servingfiles

241

WritingtestsFoxxprovidesoutoftheboxsupportforrunningtestsagainstaninstalledserviceusingtheMochatestrunner.

TestfileshavefullaccesstotheservicecontextandallArangoDBAPIsbutlikescriptscannotdefineFoxxroutes.

Runningtests

Aninstalledservice'stestscanbeexecutedfromtheadministrativewebinterface:

1. Openthe"Services"tabofthewebinterface2. Clickontheinstalledservicetobetested3. Clickonthe"Settings"tab4. Clickontheflaskiconinthetopright5. Accepttheconfirmationdialog

Notethatrunningtestsinaproductiondatabaseisnotrecommendedandmayresultindatalossifthetestsaccessthedatabase.

Whenrunningaserviceindevelopmentmodespecialcareneedstobetakenasperformingrequeststotheservice'sownroutesaspartofthetestsuitesmayresultintestsbeingexecutedwhilethedatabaseisinaninconsistentstate,leadingtounexpectedbehaviour.

TestfilepathsInordertotellFoxxaboutfilescontainingtestsuites,oneormorepatternsneedtobespecifiedinthetestsoptionoftheservicemanifest:

{

"tests":[

"**/test_*.js",

"**/*_test.js"

]

}

Thesepatternscanbeeitherrelativefilepathsor"globstar"patternswhere

*matcheszeroormorecharactersinafilename**matcheszeroormorenesteddirectories

Forexample,giventhefollowingdirectorystructure:

++test/

|++a/

||+-a1.js

||+-a2.js

||+-test.js

|+-b.js

|+-c.js

|+-d_test.js

+-e_test.js

+-test.js

Thefollowingpatternswouldmatchthefollowingfiles:

test.js:

test.js

test/*.js:

/test/b.js

/test/c.js

/test/d_test.js

Writingtests

242

https://mochajs.org

test/**/*.js:

/test/a/a1.js

/test/a/a2.js

/test/a/test.js

/test/b.js

/test/c.js

/test/d_test.js

**/test.js:

/test/a/test.js

**/*test.js:

/test/a/test.js

/test/d_test.js

/e_test.js

/test.js

Evenifmultiplepatternsmatchthesamefilethetestsinthatfilewillonlyberunonce.

Theorderoftestsisalwaysdeterminedbythefilepaths,nottheorderinwhichtheyarematchedorspecifiedinthemanifest.

TeststructureMochatestsuitescanbedefinedusingoneofthreeinterfaces:BDD,TDDorExports.

TheQUnitinterfaceofMochaisnotsupportedinArangoDB.

LikeallArangoDBcode,testcodeisalwayssynchronous.

BDDinterface

TheBDDinterfacedefinestestsuitesusingthedescribefunctionandeachtestcaseisdefinedusingtheitfunction:

'usestrict';

constassert=require('assert');

consttrueThing=true;

describe('Truethings',()=>{

it('aretrue',()=>{

assert.equal(trueThing,true);

});

});

TheBDDinterfacealsooffersthealiascontextfordescribeandspecifyforit.

Testfixturescanbehandledusingbeforeandafterforsuite-widefixturesandbeforeEachandafterEachforper-testfixtures:

describe('Falsethings',()=>{

letfalseThing;

before(()=>{

falseThing=!true;

});

it('arefalse',()=>{

assert.equal(falseThing,false);

});

});

TDDinterface

TheTDDinterfacedefinestestsuitesusingthesuitefunctionandeachtestcaseisdefinedusingthetestfunction:

'usestrict';



suite('Truethings',()=>{

test('aretrue',()=>{

Writingtests

243


});

});

TestfixturescanbehandledusingsuiteSetupandsuiteTeardownforsuite-widefixturesandsetupandteardownforper-testfixtures:

suite('Falsethings',()=>{

letfalseThing;

suiteSetup(()=>{

falseThing=!true;

});

test('arefalse',()=>{


});

});

Exportsinterface

TheExportsinterfacedefinestestcasesasmethodsofplainobjectpropertiesofthemodule.exportsobject:

'usestrict';



exports['Truethings']={

'aretrue':function(){


}

};

Thekeysbefore,after,beforeEachandafterEacharespecial-casedandbehavelikethecorrespondingfunctionsintheBDDinterface:

letfalseThing;

exports['Falsethings']={

before(){

falseThing=false;

},

'arefalse':function(){


}

};

Assertions

ArangoDBprovidestwobundledmodulestodefineassertions:

assertcorrespondstotheNode.jsassertmodule,providinglow-levelassertionsthatcanoptionallyspecifyanerrormessage.

chaiisthepopularChaiAssertionLibrary,providingbothBDDandTDDstyleassertionsusingafamiliarsyntax.

Writingtests

244

http://nodejs.org/api/assert.html

http://chaijs.com

Cross-OriginResourceSharing(CORS)TouseCORSinyourFoxxservicesyoufirstneedtoconfigureArangoDBforCORS.Asof3.2Foxxwillthenautomaticallywhitelistallresponseheadersastheyareused.

IfyouwantmorecontroloverthewhitelistorareusinganolderversionofArangoDByoucansetthefollowingresponseheadersinyourrequesthandler:

access-control-expose-headers:acomma-separatedlistofresponseheaders.Thisdefaultstoalistofallheaderstheresponseisactuallyusing(butnotincludinganyaccess-controlheaders).

access-control-allow-credentials:canbesetto"false"toforbidexposingcookies.ThedefaultvaluedependsonwhetherArangoDBtruststheorigin.Seethenotesonhttp.trusted-origin.

NotethatitisnotpossibletooverridetheseheadersfortheCORSpreflightresponse.Itisthereforenotpossibletoacceptcredentialsorcookiesonlyforindividualroutes,servicesordatabases.TheoriginneedstobetrustedaccordingtothegeneralArangoDBconfiguration(seeabove).

CrossOrigin

245

FoxxscriptsandqueuedjobsFoxxletsyoudefinescriptsthatcanbeexecutedaspartoftheinstallationandremovalprocess,invokedmanuallyorscheduledtorunatalatertimeusingthejobqueue.

Toregisteryourscript,justaddascriptssectiontoyourservicemanifest:

{

...

"scripts":{

"setup":"scripts/setup.js",

"send-mail":"scripts/send-mail.js"

}

...

}

Thescriptsyoudefineinyourservicemanifestcanbeinvokedfromthewebinterfaceintheservice'ssettingspagewiththeScriptsdropdown.

Youcanalsousethescriptsasqueuedjobs:

'usestrict';

constqueues=require('@arangodb/foxx/queues');

queues.get('default').push(

{mount:'/my-service-mount-point',name:'send-mail'},

{to:'[email protected]',body:'Hello'}

);

ScriptargumentsandreturnvaluesIfthescriptwasinvokedwithanyarguments,youcanaccessthemusingthemodule.context.argvarray.

Toreturndatafromyourscript,youcanassignthedatatomodule.exportsasusual.PleasenotethatthisdatawillbeconvertedtoJSON.

Anyerrorsraisedbythescriptwillbehandleddependingonhowthescriptwasinvoked:

ifthescriptwasinvokedfromtheHTTPAPI(e.g.usingthewebinterface),itwillreturnanerrorresponseusingtheexception'sstatusCodepropertyifspecifiedor500.ifthescriptwasinvokedfromaFoxxjobqueue,thejob'sfailurecounterwillbeincrementedandthejobwillberescheduledormarkedasfailedifnoattemptsremain.

Examples

Let'ssayyouwanttodefineascriptthattakestwonumericvaluesandreturnstheresultofmultiplyingthem:

'usestrict';


constargv=module.context.argv;

assert.equal(argv.length,2,'Expectedexactlytwoarguments');

assert.equal(typeofargv[0],'number','Expectedfirstargumenttobeanumber');

assert.equal(typeofargv[1],'number','Expectedsecondargumenttobeanumber');

module.exports=argv[0]*argv[1];

LifecycleScriptsFoxxrecognizeslifecyclescriptsiftheyaredefinedandwillinvokethemduringtheinstallation,updateandremovalprocessoftheserviceifyouwant.


246

Thefollowingscriptsarecurrentlyrecognizedaslifecyclescriptsbytheirname:"setup"and"teardown".

SetupScript

ThesetupscriptwillbeexecutedwithoutargumentsduringtheinstallationofyourFoxxservice.

Thesetupscriptmaybeexecutedmorethanonceandshouldthereforebetreatedasreentrant.Runningthesamesetupscriptagainshouldnotresultinanyerrorsorduplicatedata.

Thesetupscriptistypicallyusedtocreatecollectionsyourserviceneedsorinsertseeddatalikeinitialadministrativeuseraccountsandsoon.

Examples

'usestrict';


consttextsCollectionName=module.context.collectionName('texts');

//`textsCollectionName`isnowtheprefixednameofthisservice's"texts"collection.

//e.g."example_texts"iftheservicehasbeenmountedat`/example`

if(db._collection(textsCollectionName)===null){

constcollection=db._create(textsCollectionName);

collection.save({text:'entry1fromcollectiontexts'});



}else{

console.debug(`collection${texts}alreadyexists.Leavingituntouched.`);

}

TeardownScript

TheteardownscriptwillbeexecutedwithoutargumentsduringtheremovalofyourFoxxservice.

Itcanalsooptionallybeexecutedbeforeupgradinganservice.

Thisscripttypicallyremovesthecollectionsand/ordocumentscreatedbyyourservice'ssetupscript.

Examples

'usestrict';


consttextsCollection=module.context.collection('texts');

if(textsCollection){

textsCollection.drop();

}

Queuesconstqueues=require('@arangodb/foxx/queues')

Foxxallowsdefiningjobqueuesthatletyouperformsloworexpensiveactionsasynchronously.Thesequeuescanbeusedtosende-mails,callexternalAPIsorperformotheractionsthatyoudonotwanttoperformdirectlyorwanttoretryonfailure.

enableordisabletheFoxxqueuesfeature--foxx.queuesflag

Iftrue,theFoxxqueueswillbeavailableandjobsinthequeueswillbeexecutedasynchronously.

Thedefaultistrue.Whensettofalsethequeuemanagerwillbedisabledandanyjobsarepreventedfrombeingprocessed,whichmayreduceCPUloadabit.PleasenotethatFoxxjobqueuesaredatabase-specific.Queuesandjobsarealwaysrelativetothedatabaseinwhichtheyarecreatedoraccessed.

pollintervalforFoxxqueues--foxx.queues-poll-intervalvalue


247

ThepollintervalfortheFoxxqueuesmanager.Thevalueisspecifiedinseconds.LowervalueswillmeanmoreimmediateandmorefrequentFoxxqueuejobexecution,butwillmakethequeuethreadwakeupandquerythequeuesmoreoften.Whensettoalowvalue,thequeuethreadmightcauseCPUload.

Thedefaultis1second.IfFoxxqueuesarenotusedmuch,thenthisvaluemaybeincreasedtomakethequeuesthreadwakeupless.Forthelow-levelfunctionalityseethechapteronthetaskmanagementmodule.

Creatingorupdatingaqueue

queues.create(name,[maxWorkers]):Queue

Returnsthequeueforthegivenname.Ifthequeuedoesnotexist,anewqueuewiththegivennamewillbecreated.IfaqueuewiththegivennamealreadyexistsandmaxWorkersisset,thequeue'smaximumnumberofworkerswillbeupdated.Thequeuewillbecreatedinthecurrentdatabase.

Arguments

name:string

Nameofthequeuetocreate.

maxWorkers:number(Default:1)

Themaximumnumberofworkers.

Examples

//Createaqueuewiththedefaultnumberofworkers(i.e.one)

constqueue1=queues.create("my-queue");

//Createaqueuewithagivennumberofworkers

constqueue2=queues.create("another-queue",2);

//Updatethenumberofworkersofanexistingqueue

constqueue3=queues.create("my-queue",10);

//queue1andqueue3refertothesamequeue

assertEqual(queue1,queue3);

Fetchinganexistingqueue

queues.get(name):Queue

Returnsthequeueforthegivenname.Ifthequeuedoesnotexistanexceptionisthrowninstead.

Thequeuewillbelookedupinthecurrentdatabase.

Arguments

name:string

Nameofthequeuetofetch.

Examples

Ifthequeuedoesnotyetexistanexceptionisthrown:

queues.get("some-queue");

//Error:Queuedoesnotexist:some-queue

//at...

Otherwisethequeuewillbereturned:

constqueue1=queues.create("some-queue");

constqueue2=queues.get("some-queue");

assertEqual(queue1,queue2);

Deletingaqueuequeues.delete(name):boolean


248

Returnstrueifthequeuewasdeletedsuccessfully.Ifthequeuedidnotexist,itreturnsfalseinstead.Thequeuewillbelookedupanddeletedinthecurrentdatabase.

Whenaqueueisdeleted,jobsonthatqueuewillnolongerbeexecuted.

Deletingaqueuewillnotdeleteanyjobsonthatqueue.

Arguments

name:string

Nameofthequeuetodelete.

Examples

constqueue=queues.create("my-queue");

queues.delete("my-queue");//true

queues.delete("my-queue");//false

Addingajobtoaqueue

queue.push(script,data,[opts]):string

Thejobwillbeaddedtothespecifiedqueueinthecurrentdatabase.

Returnsthejobid.

Arguments

script:object

Ajobtypedefinition,consistingofanobjectwiththefollowingproperties:

name:string

Nameofthescriptthatwillbeinvoked.

mount:string

Mountpathoftheservicethatdefinesthescript.

backOff:Function|number(Default:1000)

Eitherafunctionthattakesthenumberoftimesthejobhasfailedbeforeasinputandreturnsthenumberofmillisecondstowaitbeforetryingthejobagain,orthedelaytobeusedtocalculateanexponentialback-off,or0fornodelay.

maxFailures:number|Infinity(Default:0):

Numberoftimesasinglerunofajobwillbere-triedbeforeitismarkedas"failed".AnegativevalueorInfinitymeansthatthejobwillbere-triedonfailureindefinitely.


Schematovalidateajob'sdataagainstbeforeenqueuingthejob.

preprocess:Function(optional)

Functiontopre-processajob's(validated)databeforeserializingitinthequeue.

data:any

Jobdataofthejob;mustbeserializabletoJSON.

opts:object(optional)

Objectwithanyofthefollowingproperties:

success:Function(optional)

Functiontobecalledafterthejobhasbeencompletedsuccessfully.


249

https://en.wikipedia.org/wiki/Exponential_backoff

failure:Function(optional)

Functiontobecalledafterthejobhasfailedtoomanytimes.

delayUntil:number|Date(Default:Date.now())

Timestampinmilliseconds(orDateinstance)untilwhichtheexecutionofthejobshouldbedelayed.

backOff:Function|number(Default:1000)

Seescript.backOff.

maxFailures:number|Infinity(Default:0):

Seescript.maxFailures.

repeatTimes:number|Function(Default:0)

Ifsettoapositivenumber,thejobwillberepeatedthismanytimes(notcountingrecoverywhenusingmaxFailures).IfsettoanegativenumberorInfinity,thejobwillberepeatedindefinitely.Ifsetto0thejobwillnotberepeated.

repeatUntil:number|Date(optional)

Ifthejobissettoautomaticallyrepeat,thiscanbesettoatimestampinmilliseconds(orDateinstance)afterwhichthejobwillnolongerrepeat.Settingthisvaluetozero,anegativevalueorInfinityhasnoeffect.

repeatDelay:number(Default:0)

Ifthejobissettoautomaticallyrepeat,thiscanbesettoanon-negativevaluetosetthenumberofmillisecondsforwhichthejobwillbedelayedbeforeitisstartedagain.

NotethatifyoupassafunctionforthebackOffcalculation,successcallbackorfailurecallbackoptionsthefunctionwillbeserializedtothedatabaseasastringandthereforemustnotrelyonanyexternalscopeorexternalvariables.

Whenthejobissettoautomaticallyrepeat,thefailurecallbackwillonlybeexecutedwhenarunofthejobhasfailedmorethanmaxFailurestimes.NotethatifthejobfailsandmaxFailuresisset,itwillberescheduledaccordingtothebackOffuntilithaseitherfailedtoomanytimesorcompletedsuccessfullybeforebeingscheduledaccordingtotherepeatDelayagain.RecoveryattemptsbymaxFailuresdonotcounttowardsrepeatTimes.

Thesuccessandfailurecallbacksreceivethefollowingarguments:

result:any

Thereturnvalueofthescriptforthecurrentrunofthejob.

jobData:any

Thedatapassedtothismethod.

job:object

ArangoDBdocumentrepresentingthejob'scurrentstate.

Examples

Let'ssaywehaveanservicemountedat/mailerthatprovidesascriptcalledsend-mail:

'usestrict';


constqueue=queues.create('my-queue');

queue.push(

{mount:'/mailer',name:'send-mail'},

{to:'[email protected]',body:'Helloworld'}

);

Thiswillnotwork,becauselogwasdefinedoutsidethecallbackfunction(thecallbackmustbeserializabletoastring):

//WARNING:THISDOESNOTWORK!

'usestrict';



250


constlog=require('console').log;//outsidethecallback'sfunctionscope

queue.push(


{to:'[email protected]',body:'Helloworld'},

{success:function(){

log('Yay!');//throws'logisnotdefined'

}}

);

Here'sanexampleofajobthatwillbeexecutedevery5secondsuntiltomorrow:

'usestrict';

constqueues=require('@arangodb/foxx').queues;


queue.push(


{to:'[email protected]',body:'Helloworld'},

{

repeatTimes:Infinity,

repeatUntil:Date.now()+(24*60*60*1000),

repeatDelay:5*1000

}

);

Fetchingajobfromthequeue

queue.get(jobId):Job

Createsaproxyobjectrepresentingajobwiththegivenjobid.

Thejobwillbelookedupinthespecifiedqueueinthecurrentdatabase.

ReturnsthejobforthegivenjobId.Propertiesofthejobobjectwillbefetchedwhenevertheyarereferencedandcannotbemodified.

Arguments

jobId:string

Theidofthejobtocreateaproxyobjectfor.

Examples

constjobId=queue.push({mount:'/logger',name:'log'},'HelloWorld!');

constjob=queue.get(jobId);

assertEqual(job.id,jobId);

Deletingajobfromthequeue

queue.delete(jobId):boolean

Deletesajobwiththegivenjobid.Thejobwillbelookedupanddeletedinthespecifiedqueueinthecurrentdatabase.

Arguments

jobId:string

Theidofthejobtodelete.

Returnstrueifthejobwasdeletedsuccessfully.Ifthejobdidnotexistitreturnsfalseinstead.

Fetchinganarrayofjobsinaqueue

Examples

constlogScript={mount:'/logger',name:'log'};

queue.push(logScript,'HelloWorld!',{delayUntil:Date.now()+50});

assertEqual(queue.pending(logScript).length,1);

//50mslater...


251

assertEqual(queue.pending(logScript).length,0);

assertEqual(queue.progress(logScript).length,1);

//evenlater...

assertEqual(queue.progress(logScript).length,0);

assertEqual(queue.complete(logScript).length,1);

Fetchinganarrayofpendingjobsinaqueuequeue.pending([script]):Array<string>

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"pending",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments

script:object(optional)


name:stringNameofthescript.

mount:stringMountpathoftheservicedefiningthescript.

Fetchinganarrayofjobsthatarecurrentlyinprogress

queue.progress([script])

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"progress",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments





Fetchinganarrayofcompletedjobsinaqueuequeue.complete([script]):Array<string>

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"complete",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments





Fetchinganarrayoffailedjobsinaqueue

queue.failed([script]):Array<string>


252

Returnsanarrayofjobidsofjobsinthegivenqueuewiththestatus"failed",optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments





Fetchinganarrayofalljobsinaqueue

queue.all([script]):Array<string>

Returnsanarrayofjobidsofalljobsinthegivenqueue,optionallyfilteredbythegivenjobtype.Thejobswillbelookedupinthespecifiedqueueinthecurrentdatabase.

Arguments





Abortingajob

job.abort():void

Abortsanon-completedjob.

Setsajob'sstatusto"failed"ifitisnotalready"complete",withoutcallingthejob'sonFailurecallback.


253

Migrating2.xservicesto3.0WhenmigratingservicesfromolderversionsofArangoDBitisgenerallyrecommendedyoumakesuretheyworkinlegacycompatibilitymode,whichcanalsoserveasastop-gapsolution.

ThischapteroutlinesthemajordifferencesintheFoxxAPIbetweenArangoDB2.8andArangoDB3.0.

Generalchanges

TheconsoleobjectinlaterversionsofArangoDB2.ximplementedaspecialFoxxconsoleAPIandwouldoptionallylogmessagestoacollection.ArangoDB3.0restorestheoriginalbehaviourwhereconsoleisthesameobjectavailablefromtheconsolemodule.

Migrating2.xservices

254

Migratingfrompre-2.8WhenmigratingfromaversionolderthanArangoDB2.8pleasenotethatstartingwithArangoDB2.8thebehaviouroftherequirefunctionmorecloselymimicsthebehaviourobservedinNode.jsandmodulebundlersforbrowsers,e.g.:

Inafile/routes/examples.js(relativetotherootfolderoftheservice):

require('./my-module')willbeattemptedtoberesolvedinthefollowingorder:

1. /routes/my-module(relativetoserviceroot)2. /routes/my-module.js(relativetoserviceroot)3. /routes/my-module.json(relativetoserviceroot)4. /routes/my-module/index.js(relativetoserviceroot)5. /routes/my-module/index.json(relativetoserviceroot)

require('lodash')willbeattemptedtoberesolvedinthefollowingorder:

1. /routes/node_modules/lodash(relativetoserviceroot)2. /node_modules/lodash(relativetoserviceroot)3. ArangoDBmodulelodash4. Nodecompatibilitymodulelodash5. BundledNPMmodulelodash

require('/abs/path')willbeattemptedtoberesolvedinthefollowingorder:

1. /abs/path(relativetofilesystemroot)2. /abs/path.js(relativetofilesystemroot)3. /abs/path.json(relativetofilesystemroot)4. /abs/path/index.js(relativetofilesystemroot)5. /abs/path/index.json(relativetofilesystemroot)

ThisbehaviourisincompatiblewiththesourcecodegeneratedbytheFoxxgeneratorinthewebinterfacebeforeArangoDB2.8.

Note:Theorg/arangodbmoduleisaliasedtothenewname@arangodbinArangoDB3.0.0andthe@arangodbmodulewasaliasedtotheoldnameorg/arangodbinArangoDB2.8.0.Eitheronewillworkin2.8and3.0butoutsideoflegacyservicesyoushoulduse@arangodbgoingforward.

FoxxqueueInArangoDB2.6FoxxintroducedanewwaytodefinequeuedjobsusingFoxxscriptstoreplacethefunction-basedjobtypedefinitionswhichwerecausingproblemswhenrestartingtheserver.Thefunction-basedjobshavebeenremovedin2.7andarenolongersupportedatall.

CoffeeScriptArangoDB3.0nolongerprovidesbuilt-insupportforCoffeeScriptsourcefiles,eveninlegacycompatibilitymode.IfyouwanttouseanalternativelanguagelikeCoffeeScript,makesuretopre-compiletherawsourcefilestoJavaScriptandusethecompiledJavaScriptfilesintheservice.

TherequestmoduleThe@arangodb/requestmodulewhenusedwiththejsonoptionpreviouslyoverwrotethestringinthebodypropertyoftheresponseobjectoftheresponsewiththeparsedJSONbody.In2.8thiswaschangedsotheparsedJSONbodyisaddedasthejsonpropertyoftheresponseobjectinadditiontooverwritingthebodyproperty.In3.0andlater(includinglegacycompatibilitymode)thebodypropertyisnolongeroverwrittenandmustusethejsonpropertyinstead.Notethatthisonlyaffectscodeusingthejsonoptionwhenmakingtherequest.


255

BundledNPMmodules

ThebundledNPMmoduleshavebeenupgradedandmayincludebackwards-incompatiblechanges,especiallytheAPIofjoihaschangedseveraltimes.Ifindoubtyoushouldbundleyourownversionsofthesemodulestoensurespecificversionswillbeused.

Theutilitymodulelodashisnowavailableandshouldbeusedinsteadofunderscore,butbothmoduleswillcontinuetobeprovided.


256

ManifestManyofthefieldsthatwererequiredinArangoDB2.xarenowoptionalandcanbesafelyomitted.

ToavoidcompatibilityproblemswithfutureversionsofArangoDByoushouldalwaysspecifytheenginesfield,e.g.:

{

"engines":{

"arangodb":"^3.0.0"

}

}

Controllers&exports

PreviouslyFoxxdistinguishedbetweenexportsandcontrollers,eachofwhichcouldbespecifiedasanobject.InArangoDB3.0thesehavebeenmergedintoasinglemainfieldspecifyinganentryfile.

Theeasiestwaytomigrateservicesusingmultipleexportsand/orcontrollersistocreateaseparateentryfilethatimportsthesefiles:

Old(manifest.json):

{

"exports":{

"doodads":"doodads.js",

"dingbats":"dingbats.js"

},

"controllers":{

"/doodads":"routes/doodads.js",

"/dingbats":"routes/dingbats.js",

"/":"routes/root.js"

}

}

New(manifest.json):

{

"main":"index.js"

}

New(index.js):

'usestrict';

module.context.use('/doodads',require('./routes/doodads'));

module.context.use('/dingbats',require('./routes/dingbats'));

module.context.use('/',require('./routes/root'));

module.exports={

doodads:require('./doodads'),

dingbats:require('./dingbats')

};

IndexredirectIfyoupreviouslydidnotdefinethedefaultDocumentfield,pleasenotethatinArangoDB3.0thefieldwillnolongerdefaulttothevalueindex.htmlwhenomitted:

Old:

{

//nodefaultDocument

}

manifest.json

257

New:

{

"defaultDocument":"index.html"

}

ThisalsomeansitisnolongernecessarytospecifythedefaultDocumentfieldwithanemptyvaluetopreventtheredirectandbeabletoserverequestsatthe/(root)pathofthemountpoint:

Old:

{

"defaultDocument":""

}

New:

{

//nodefaultDocument

}

AssetsTheassetsfieldisnolongersupportedinArangoDB3.0outsideoflegacycompatibilitymode.

Ifyoupreviouslyusedthefieldtoserveindividualfilesas-isyoucansimplyusethefilesfieldinstead:

Old:

{

"assets":{

"client.js":{

"files":["assets/client.js"],

"contentType":"application/javascript"

}

}

}

New:

{

"files":{

"client.js":{

"path":"assets/client.js",

"type":"application/javascript"

}

}

}

Ifyoureliedonbeingabletospecifymultiplefilesthatshouldbeconcatenated,youwillhavetousebuildtoolsoutsideofArangoDBtopreparethesefilesaccordingly.

RootelementTherootElementfieldisnolongersupportedandhasbeenremovedentirely.

IfyourcontrollersreliedonthisfieldbeingavailableyouneedtoadjustyourschemasandroutestobeabletohandlethefullJSONstructureofincomingdocuments.

Systemservices

manifest.json

258

TheisSystemfieldisnolongersupported.ThepresenceorabsenceofthefieldhadnoeffectinmostrecentversionsofArangoDB2.xandhasnowbeenremovedentirely.

manifest.json

259

TheapplicationcontextTheglobalapplicationContextvariableavailableinFoxxmoduleshasbeenreplacedwiththecontextattributeofthemodulevariable.Forconsistencyitisnowreferredtoastheservicecontextthroughoutthisdocumentation.

SomemethodsoftheservicecontexthavechangedinArangoDB3.0:

fileName()nowbehaveslikepath()didinArangoDB2.xpath()hasbeenremoved(usefileName()instead)foxxFileName()hasbeenremoved(usefileName()instead)

Additionallytheversionandnameattributeshavebeenremovedandcannowonlybeaccessedviathemanifestattribute(asmanifest.versionandmanifest.name).Notethatthecorrespondingmanifestfieldsarenowoptionalandmaybeomitted.

Theoptionsattributehasalsobeenremovedasitshouldbeconsideredanimplementationdetail.Youshouldinsteadaccessthedependenciesandconfigurationattributesdirectly.

Theinternal_prefixattribute(whichwasanaliasforbasePath)andtheinternalcommentandclearCommentsmethods(whichwereusedbythemagicaldocumentationcommentsinArangoDB2.x)havealsobeenremoved.

Theinternal_serviceattribute(whichprovidesaccesstotheserviceitself)hasbeenrenamedtoservice.

applicationContext

260

RepositoriesandmodelsPreviouslyFoxxwasheavilybuiltaroundtheconceptofrepositoriesandmodels,whichprovidedcomplexbutrarelynecessaryabstractionsontopofArangoDBcollectionsanddocuments.InArangoDB3.0thesehavebeenremovedentirely.

Repositoriesvscollections

RepositoriesmostlywrappedmethodsthatalreadyexistedonArangoDBcollectionobjectsandprimarilydealtwithconvertingbetweenplainArangoDBdocumentsandFoxxmodelinstances.InArangoDB3.0youcansimplyusethesecollectionsdirectlyandtreatdocumentsasplainJavaScriptobjects.

Old:

'usestrict';

constFoxx=require('org/arangodb/foxx');

constmyRepo=newFoxx.Repository(

applicationContext.collection('myCollection'),

{model:Foxx.Model}

);

//...

constmodels=myRepo.byExample({color:'green'});

res.json(models.map(function(model){

returnmodel.forClient();

}));

New:

'usestrict';

constmyDocs=module.context.collection('myCollection');

//...

constdocs=myDocs.byExample({color:'green'});

res.json(docs);

Schemavalidation

ThemainpurposeofmodelsinArangoDB2.xwastovalidateincomingdatausingjoischemas.InmorerecentversionsofArangoDB2.xitwasalreadypossibletopasstheseschemasdirectlyinmostplaceswhereamodelwasexpectedasanargument.Theonlydifferenceisthatschemasshouldnowbeconsideredthedefault.

IfyoupreviouslyreliedontheautomaticvalidationofFoxxmodelinstanceswhensettingattributesorinstantiatingmodelsfromuntrusteddata,youcansimplyusetheschema'svalidatemethoddirectly.

Old:

'usestrict';


constmySchema={


size:joi.number().required()

};


constMyModel=Foxx.Model.extend({schema:mySchema});

//...

constmodel=newMyModel(req.json());

if(!model.isValid){

res.status(400);


261

res.write('Badrequest');

return;

}

New:

'usestrict';


//Notethisisnowwrappedinajoi.object()

constmySchema=joi.object({



}).required();

//...

constresult=mySchema.validate(req.body);

if(result.errors){

res.status(400);

res.write('Badrequest');

return;

}

Migratingmodels

Whilemostusecasesformodelscannowbereplacedwithplainjoischemas,thereisstilltheconceptofa"model"inFoxxinArangoDB3.0althoughitisquitedifferentfromFoxxmodelsinArangoDB2.x.

AmodelinFoxxnowreferstoaplainJavaScriptobjectwithanoptionalschemaattributeandtheoptionalmethodsforClientandfromClient.Modelscanbeusedinsteadofplainjoischemastodefinerequestandresponsebodiesbuttherearenomodel"instances"inArangoDB3.0.

Old:

'usestrict';

const_=require('underscore');



constMyModel=Foxx.Model.extend({

schema:{



},

forClient(){

return_.omit(this.attributes,['_key','_id','_rev']);

}

});

//...

ctrl.get(/*...*/)

.bodyParam('body',{type:MyModel});

New:

'usestrict';

const_=require('lodash');


constMyModel={

schema:joi.object({



}).required(),

forClient(data){

return_.omit(data,['_key','_id','_rev']);

}

};


262

//...

router.get(/*...*/)

.body(MyModel);

TriggersWhensaving,updating,replacingordeletingmodelsinArangoDB2.xusingtherepositorymethodstherepositoryandmodelwouldfireeventsthatcouldbesubscribedtoinordertoperformside-effects.

Notethatevenin2.xtheseeventswouldnotfirewhenusingqueriesormanipulatingdocumentsinanyotherwaythanusingthespecificrepositorymethodsthatoperatedonindividualdocuments.

ThisbehaviourisnolongeravailableinArangoDB3.0butcanbeemulatedbyusinganEventEmitterdirectlyifitisnotpossibletosolvetheproblemdifferently:

Old:

'usestrict';


constMyModel=Foxx.Model.extend({

//...

},{

afterRemove(){

console.log(this.get('name'),'wasremoved');

}

});

//...

constmodel=myRepo.firstExample({name:'myName'});

myRepo.remove(model);

//->"myNamewasremovedsuccessfully"

New:

'usestrict';

constEventEmitter=require('events');

constemitter=newEventEmitter();

emitter.on('afterRemove',function(doc){

console.log(doc.name,'wasremoved');

});

//...

constdoc=myDocs.firstExample({name:'myName'});

myDocs.remove(doc);

emitter.emit('afterRemove',doc);


Orsimply:

'usestrict';

functionafterRemove(doc){

console.log(doc.name,'wasremoved');

}

//...

constdoc=myDocs.firstExample({name:'myName'});

myDocs.remove(doc);

afterRemove(doc);



263

ControllersvsroutersFoxxControllershavebeenreplacedwithrouters.Thisismorethanacosmeticchangeastherearesignificantdifferencesinbehaviour:

Controllerswereautomaticallymountedwhenthefiledefiningthemwasexecuted.Routersneedtobeexplicitlymountedusingthemodule.context.usemethod.Routerscanalsobeexported,importedandevennested.Thismakesiteasiertosplitupcomplexroutingtreesacrossmultiplefiles.

Old:

'usestrict';


constctrl=newFoxx.Controller(applicationContext);

ctrl.get('/hello',function(req,res){

//...

});

New:

'usestrict';

constcreateRouter=require('org/arangodb/foxx/router');


//Ifyouareimportingthisfilefromyourentryfile("main"):

module.exports=router;

//Otherwise:module.context.use(router);

router.get('/hello',function(req,res){

//...

});

Somegeneralchangesinbehaviourthatmighttripyouup:

WhenspecifyingpathparameterswithschemasFoxxwillnowignoretherouteiftheschemadoesnotmatch(i.e./hello/foxxwillnolongermatch/hello/:numifnumspecifiesaschemathatdoesn'tmatchthevalue"foxx").Withcontrollersthiscouldpreviouslyresultinusersseeinga400(badrequest)errorwhentheyshouldinsteadbeserveda404(notfound)response.

WhenarequestismadewithanHTTPverbnotsupportedbyanendpoint,Foxxwillnowrespondwitha405(methodnotallowed)errorwithanappropriateAllowedheaderlistingthesupportedHTTPverbsforthatendpoint.

FoxxwillnolongerparseyourJSDoccommentstogenerateroutedocumentation(usethesummaryanddescriptionmethodsoftheendpointinstead).

TheapiDocumentationmethodnowlivesontheservicecontextandbehavesslightlydifferently.

ThereisnorouterequivalentfortheactivateAuthenticationandactivateSessionsmethods.Insteadyoushouldusethesessionmiddleware(seethesectiononsessionsbelow).

Thereisnodelaliasforthedeletemethodonrouters.IthasalwaysbeensafetousekeywordsasmethodnamesinFoxx,sotheuseofthisaliaswasalreadydiscouragedbefore.

TheallRoutesproxyisnolotavailableonroutersbutcaneasilybereplacedwithmiddlewareorchildrouters.

Controllers

264

TherequestcontextWhendefiningarouteonacontrollerthecontrollerwouldreturnanobjectcalledrequestcontext.Routersreturnasimilarobjectcalledendpoint.Routersalsoreturnendpointswhenmountingchildroutersormiddleware,asdoestheusemethodoftheservicecontext.

ThemaindifferencesbetweenthenewendpointsandtheobjectsreturnedbycontrollersinpreviousversionsofArangoDBare:

bodyParamisnowsimplycalledbody;itisnolongerneccessaryorpossibletogivethebodyanameandtherequestbodywillnotshowupintherequestparameters.It'salsopossibletospecifyaMIMEtype

body,queryParamandpathParamnowtakepositionargumentsinsteadofanobject.Forspecificsseetheendpointdocumentation.

notesisnowcalleddescriptionandtakesasinglestringargument.

onlyIfandonlyIfAuthenticatedarenolongeravailable;theycanbeemulatedwithmiddlewareifnecessary:

Old:

ctrl.get(/*...*/)

.onlyIf(function(req){

if(!req.user){

thrownewError('Notauthenticated!');

}

});

New:

router.use(function(req,res,next){


res.throw(403,'Notauthenticated!');

}

next();

});

router.get(/*...*/);

Controllers

265

ErrorhandlingTheerrorResponsemethodprovidedbycontrollerrequestcontextshasnoequivalentinrouterendpoints.Ifyouwanttohandlespecificerrortypeswithspecificstatuscodesyouneedtocatchthemexplicitly,eitherintherouteorinamiddleware:

Old:

ctrl.get('/puppies',function(req,res){

//Exceptionisthrownhere

})

.errorResponse(TooManyPuppiesError,400,'Somethingwentwrong!');

New:

ctrl.get('/puppies',function(req,res){

try{

//Exceptionisthrownhere

}catch(e){

if(!(einstanceofTooManyPuppiesError)){

throwe;

}

res.throw(400,'Somethingwentwrong!');

}

})

//The"error"methodmerelydocumentsthemeaning

//ofthestatuscodeandhasnoothereffect.

.error(400,'Throwniftherearetoomanypuppies.');

Notethaterrorscreatedwithhttp-errorsarestillhandledbyFoxxintelligently.Infactres.throwisjustahelpermethodforcreatingandthrowingtheseerrors.

Controllers

266

Before,afterandaroundThebefore,afterandaroundmethodscaneasilybereplacedbymiddleware:

Old:

letstart;

ctrl.before(function(req,res){

start=Date.now();

});

ctrl.after(function(req,res){

console.log('Requesthandledin',(Date.now()-start),'ms');

});

New:


letstart=Date.now();

next();

console.log('Requesthandledin',(Date.now()-start),'ms');

});

Notethatunlikearoundmiddlewarereceivesthenextfunctionasthethirdargument(the"opts"argumenthasnoequivalent).

Controllers

267

RequestobjectsThenamesofsomeattributesoftherequestobjecthavebeenadjustedtomorecloselyalignwiththoseofthecorrespondingmethodsontheendpointobjectsandestablishedconventionsinotherJavaScriptframeworks:

req.urlParametersisnowcalledreq.pathParams

req.parametersisnowcalledreq.queryParams

req.params()isnowcalledreq.param()

req.requestTypeisnowcalledreq.method

req.compatibilityisnowcalledreq.arangoVersion

req.userisnowcalledreq.arangoUser

Someattributeshavebeenremovedorchanged:

req.cookieshasbeenremovedentirely(usereq.cookie(name))

req.requestBodyhasbeenremovedentirely(seebelow)

req.suffixisnowastringratherthananarray

Additionallythereq.serverandreq.clientattributesarenolongeravailable.Theinformationisnowexposedinawaythatcan(optionally)transparentlyhandleproxyforwardingheaders:

req.hostnamedefaultstoreq.server.address

req.portdefaultstoreq.server.port

req.remoteAddressdefaultstoclient.address

req.remotePortdefaultstoclient.port

Finally,thereq.cookiemethodnowtakesthesignedoptionsdirectly.

Old:

constsid=req.cookie('sid',{

signed:{

secret:'keyboardcat',

algorithm:'sha256'

}

});

New:

constsid=req.cookie('sid',{


algorithm:'sha256'

});

RequestbodiesThereq.bodyisnolongeramethodandnolongerautomaticallyparsesJSONrequestbodiesunlessarequestbodywasdefined.Thereq.rawBodynowcorrespondstothereq.rawBodyBufferofArangoDB2.xandisalsonolongeramethod.

Old:

ctrl.post('/',function(req,res){

constdata=req.body();

//...

});

Controllers

268

New:

router.post('/',function(req,res){

constdata=req.body;

//...

})

.body(['json']);

Orsimply:



constdata=req.body;

//...

})

.body(joi.object().optional());

Multipartrequests

Thereq.requestPartsmethodhasbeenremovedentirely.Ifyouneedtoacceptmultipartrequestbodies,youcansimplydefinetherequestbodyusingamultipartMIMEtypelikemultipart/form-data:

Old:

ctrl.post('/',function(req,res){

constparts=req.requestParts();

//...

});

New:


constparts=req.body;

//...

})

.body(['multipart/form-data']);

Controllers

269

ResponseobjectsTheresponseobjecthasalotofnewmethodsinArangoDB3.0butotherwiseremainssimilartotheresponseobjectofpreviousversions:

Theres.sendmethodbehavesverydifferentlyfromhowthemethodwiththesamenamebehavedinArangoDB2.x:theconversionnowtakestheresponsebodydefinitionoftherouteintoaccount.Thereisanewmethodres.writethatimplementstheoldbehaviour.

Notethatconsecutivecallstores.writewillappendtotheresponsebodyratherthanreplacingitlikeres.send.

Theres.contentTypepropertyisalsonolongeravailable.IfyouwanttosettheMIMEtypeoftheresponsebodytoanexplicitvalueyoushouldsetthecontent-typeheaderinstead:

Old:

res.contentType='application/json';

res.body=JSON.stringify(results);

New:

res.set('content-type','application/json');

res.body=JSON.stringify(results);

Orsimply:

//setsthecontenttypetoJSON

//ifithasnotalreadybeenset

res.json(results);

Theres.cookiemethodnowtakesthesignedoptionsaspartoftheregularoptionsobject.

Old:

res.cookie('sid','abcdef',{

ttl:60*60,

signed:{


algorithm:'sha256'

}

});

New:

res.cookie('sid','abcdef',{

ttl:60*60,


algorithm:'sha256'

});

Controllers

270

DependencyinjectionThereisnoequivalentoftheaddInjectormethodavailableinArangoDB2.xcontrollers.Mostusecasescanbesolvedbysimplyusingplainvariablesbutifyouneedsomethingmoreflexibleyoucanalsousemiddleware:

Old:

ctrl.addInjector('magicNumber',function(){

returnMath.random();

});

ctrl.get('/',function(req,res,injected){

res.json(injected.magicNumber);

});

New:

functionmagicMiddleware(name){

return{

register(){

letmagic;

returnfunction(req,res,next){

if(!magic){

magic=Math.random();

}

req[name]=magic;

next();

};

}

};

}

router.use(magicMiddleware('magicNumber'));


res.json(req.magicNumber);

});

Orsimply:

constmagicNumber=Math.random();


res.json(magicNumber);

});

Controllers

271

SessionsThectrl.activateSessionsmethodandtherelatedutil-sessions-localFoxxservicehavebeenreplacedwiththeFoxxsessionsmiddleware.Itisnolongerpossibletousethebuilt-insessionstoragebutyoucansimplypassinanydocumentcollectiondirectly.

Old:

constlocalSessions=applicationContext.dependencies.localSessions;

constsessionStorage=localSessions.sessionStorage;

ctrl.activateSessions({

sessionStorage:sessionStorage,

cookie:{secret:'keyboardcat'}

});

ctrl.destroySession('/logout',function(req,res){

res.json({message:'Goodbye!'});

});

New:

constsessionMiddleware=require('@arangodb/foxx/sessions');

constcookieTransport=require('@arangodb/foxx/sessions/transports/cookie');

router.use(sessionMiddleware({


transport:cookieTransport('keyboardcat')

}));

router.post('/logout',function(req,res){

req.sessionStorage.clear(req.session);

res.json({message:'Goodbye!'});

});

Sessions

272

AuthandOAuth2Theutil-simple-authandutil-oauth2FoxxserviceshavebeenreplacedwiththeFoxxauthandFoxxOAuth2modules.Itisnolongernecessarytoinstalltheseservicesasdependenciesinordertousethefunctionality.

Old:

'usestrict';

constauth=applicationContext.dependencies.simpleAuth;

//...

constvalid=auth.verifyPassword(authData,password);

New:

'usestrict';

constcreateAuth=require('@arangodb/foxx/auth');

constauth=createAuth();//Usedefaultconfiguration

//...

constvalid=auth.verifyPassword(authData,password);

AuthandOAuth2

273

FoxxqueriesThecreateQuerymethodhasbeenremoved.ItcanbetriviallyreplacedwithplainJavaScriptfunctionsanddirectcallstothedb._querymethod:

Old:

'usestrict';


constquery=Foxx.createQuery({

query:'FORuIN_usersSORTu.userASCRETURNu[@propName]',

params:['propName'],

transform:function(results,uppercase){

return(

uppercase

?results[0].toUpperCase()

:results[0].toLowerCase()

);

}

});

query('user',true);

New:

'usestrict';



functionquery(propName,uppercase){

constresults=db._query(aql`

FORuIN_users

SORTu.userASC

RETURNu[${propName}]

`);

return(

uppercase

?results[0].toUpperCase()

:results[0].toLowerCase()

);

}

query('user',true);

FoxxQueries

274

Legacycompatibilitymodefor2.8servicesArangoDB3continuestosupportFoxxserviceswrittenforArangoDB2.8byrunningtheminaspeciallegacycompatibilitymodethatprovidesaccesstosomeofthemodulesandAPIsnolongerprovidedin3.0andbeyond.

Note:LegacycompatibilitymodeisstrictlyintendedasatemporarystopgapsolutionforsupportingexistingserviceswhileupgradingtoArangoDB3.0andshouldnotbeconsideredapermanentfeatureofArangoDBorFoxx.

Inordertomarkanexistingserviceasalegacyservice,justmakesurethefollowingattributeisdefinedintheservicemanifest:

"engines":{

"arangodb":"^2.8.0"

}

ThissemanticversionrangedenotesthattheserviceisknowntoworkwithArangoDB2.8.0andsupportsallnewerversionsofArangoDBuptobutnotincluding3.0.0(noranydevelopmentversionof3.0.0andgreater).

Anysimilarversionrangethedoesnotinclude3.0.0orgreaterwillhavethesameeffect(e.g. 2.5.0willalsotriggerthelegacycompatibilitymode,aswill1.2.3,but>=2.8.0willnotasitindicatescompatibilitywithallversionsgreaterorequal2.8.0,notjustthosewithinthe2.xversionrange).

FeaturessupportedinlegacycompatibilitymodeLegacycompatibilitymodesupportstheoldmanifestformat,specifically:

mainisignoredcontrollerswillbemountedasin2.8exportswillbeexecutedasin2.8

AdditionallytheisSystemattributewillbeignoredifpresentbutdoesnotresultinawarninginlegacycompatibilitymode.

TheFoxxconsoleisavailableastheconsolepseudo-globalvariable(shadowingtheglobalconsoleobject).

TheservicecontextisavailableastheapplicationContextpseudo-globalvariableinthecontrollers,exports,scriptsandtestsasin2.8.Thefollowingadditionalpropertiesareavailableontheservicecontextinlegacycompatibilitymode:

path()isanaliasfor3.xfileName()(usingpath.jointobuildfilepaths)fileName()behavesasin2.x(usingfs.safeJointobuildfilepaths)foxxFileName()isanaliasfor2.xfileNameversionexposestheservicemanifest'sversionattributenameexposestheservicemanifest'snameattributeoptionsexposestheservice'srawoptions

Thefollowingmethodsareremovedontheservicecontextinlegacycompatibilitymode:

use()--use@arangodb/foxx/controllerinsteadapiDocumentation()--usecontroller.apiDocumentation()insteadregisterType()--notsupportedinlegacycompatibilitymode

Thefollowingmodulesthathavebeenremovedorreplacedin3.0.0areavailableinlegacycompatibilitymode:

@arangodb/foxx/authentication

@arangodb/foxx/console

@arangodb/foxx/controller

@arangodb/foxx/model

@arangodb/foxx/query

@arangodb/foxx/repository

@arangodb/foxx/schema

@arangodb/foxx/sessions

@arangodb/foxx/template_middleware


275

http://semver.org

The@arangodb/foxxmodulealsoprovidesthesameexportsasin2.8,namely:

Controllerfrom@arangodb/foxx/controllercreateQueryfrom@arangodb/foxx/queryModelfrom@arangodb/foxx/modelRepositoryfrom@arangodb/foxx/repositorytoJSONSchemafrom@arangodb/foxx/schemagetExportsandrequireAppfrom@arangodb/foxx/managerqueuesfrom@arangodb/foxx/queues

Anyfeaturenotsupportedin2.8willalsonotworkinlegacycompatibilitymode.WhenmigratingfromanolderversionofArangoDBitisagoodideatomigratetoArangoDB2.8firstforaneasierupgradepath.

AdditionallypleasenotethedifferenceslaidoutinthechapterMigratingfrompre-2.8inthemigrationguide.


276

UsermanagementFoxxdoesnotprovideanyusermanagementoutoftheboxbutitisveryeasytorollyourownsolution:

Thesessionmiddlewareprovidesmechanismsforaddingsessionlogictoyourservice,usinge.g.acollectionorJSONWebTokenstostorethesessionsbetweenrequests.

Theauthmoduleprovidesutilitiesforbasicpasswordverificationandhashing.

Thefollowingexampleservicedemonstrateshowusermanagementcanbeimplementedusingthesebasicbuildingblocks.

Settingupthecollections

Let'ssaywewanttostoresessionsandusersincollections.Wecanusethesetupscripttomakesurethesecollectionsarecreatedbeforetheserviceismounted.

Firstaddasetupscripttoyourmanifestifitisn'talreadydefined:

"scripts":{

"setup":"scripts/setup.js"

}

Thencreatethesetupscriptwiththefollowingcontent:

'usestrict';


constsessions=module.context.collectionName('sessions');

constusers=module.context.collectionName('users');

if(!db._collection(sessions)){

db._createDocumentCollection(sessions);

}

if(!db._collection(users)){

db._createDocumentCollection(users);

}

db._collection(users).ensureIndex({

type:'hash',

fields:['username'],

unique:true

});

Creatingtherouter

Thefollowingmainfiledemonstratesbasicusermanagement:

'usestrict';


constcreateAuth=require('@arangodb/foxx/auth');


constsessionsMiddleware=require('@arangodb/foxx/sessions');

constauth=createAuth();


constusers=module.context.collection('users');



transport:'cookie'

});



router.get('/whoami',function(req,res){

Usermanagement

277

try{

constuser=users.document(req.session.uid);

res.send({username:user.username});

}catch(e){

res.send({username:null});

}

})

.description('Returnsthecurrentlyactiveusername.');

router.post('/login',function(req,res){

//Thismayreturnauserobjectornull

constuser=users.firstExample({

username:req.body.username

});

constvalid=auth.verify(

//Pretendtovalidateevenifnouserwasfound

user?user.authData:{},

req.body.password

);

if(!valid)res.throw('unauthorized');

//Logtheuserin

req.session.uid=user._key;


res.send({sucess:true});

})

.body(joi.object({

username:joi.string().required(),

password:joi.string().required()

}).required(),'Credentials')

.description('Logsaregistereduserin.');

router.post('/logout',function(req,res){

if(req.session.uid){

req.session.uid=null;


}

res.send({success:true});

})

.description('Logsthecurrentuserout.');

router.post('/signup',function(req,res){

constuser=req.body;

try{

//Createanauthenticationhash

user.authData=auth.create(user.password);

deleteuser.password;

constmeta=users.save(user);

Object.assign(user,meta);

}catch(e){

//Failedtosavetheuser

//We'llassumetheUniqueConstrainthasbeenviolated

res.throw('badrequest','Usernamealreadytaken',e);

}

//Logtheuserin



res.send({success:true});

})

.body(joi.object({

username:joi.string().required(),

password:joi.string().required()

}).required(),'Credentials')

.description('Createsanewuserandlogsthemin.');

Usermanagement

278

RelatedmodulesThesearesomeofthemodulesoutsideofFoxxyouwillfindusefulwhenwritingFoxxservices.

AdditionallytherearemodulesprovidingsomelevelofcompatibilitywithNode.jsaswellasanumberofbundledNPMmodules(likelodashandjoi).FormoreinformationonthesemodulesseetheJavaScriptmodulesappendix.

The @arangodbmodulerequire('@arangodb')

ThismoduleprovidesaccesstovariousArangoDBinternalsaswellasthreeofthemostimportantexportsnecessarytoworkwiththedatabaseinFoxx:db,aqlanderrors.

YoucanfindafulldescriptionofthismoduleintheArangoDBmoduleappendix.

The @arangodb/requestmodulerequire('@arangodb/request')

ThismoduleprovidesafunctionformakingHTTPrequeststoexternalservices.Notethatwhilethisallowscommunicatingwiththird-partyservicesitmayaffectdatabaseperformancebyblockingFoxxrequestsasArangoDBwaitsfortheremoteservicetorespond.Ifyouroutinelymakerequeststoslowexternalservicesandarenotdirectlyinterestedintheresponseitisprobablyabetterideatodelegatetheactualrequest/responsecycletoagatewayservicerunningoutsideArangoDB.

Youcanfindafulldescriptionofthismoduleintherequestmoduleappendix.

The @arangodb/general-graphmodulerequire('@arangodb/general-graph')

ThismoduleprovidesaccesstoArangoDBgraphdefinitionsandvariouslow-levelgraphoperationsinJavaScript.FormorecomplexqueriesitisprobablybettertouseAQLbutthismodulecanbeusefulinyoursetupandteardownscriptstocreateanddestroygraphdefinitions.

Formoreinformationseethechapteronthegeneralgraphmodule.

Relatedmodules

279

AuthenticationconstcreateAuth=require('@arangodb/foxx/auth');

Authenticatorsallowimplementingbasicpasswordmechanismusingsimplebuilt-inhashingfunctions.

ForafullexampleofsessionswithauthenticationandregistrationseetheexampleinthechapteronUserManagement.

CreatinganauthenticatorcreateAuth([options]):Authenticator

Createsanauthenticator.

Arguments



method:string(Default:"sha256")

Thehashingalgorithmtousetocreatepasswordhashes.Theauthenticatorwillbeabletoverifypasswordsagainsthashesusinganysupportedhashingalgorithm.Thisonlyaffectsnewhashescreatedbytheauthenticator.

Supportedvalues:

"md5"

"sha1"

"sha224"

"sha256"

"sha384"

"sha512"

saltLength:number(Default:16)

Lengthofthesaltsthatwillbegeneratedforpasswordhashes.

Returnsanauthenticator.

Creatingauthenticationdataobjectsauth.create(password):AuthData

Createsanauthenticationdataobjectforthegivenpasswordwiththefollowingproperties:

method:string

Themethodusedtogeneratethehash.

salt:string

Arandomsaltusedtogeneratethishash.

hash:string

Thehashstringitself.

Arguments

password:string

Apasswordtohash.

Returnstheauthenticationdataobject.

Authentication

280

Validatingpasswordsagainstauthenticationdataobjectsauth.verify([hash,[password]]):boolean

Verifiesthegivenpasswordagainstthegivenhashusingaconstanttimestringcomparison.

Arguments

hash:AuthData(optional)

Aauthenticationdataobjectgeneratedwiththecreatemethod.

password:string(optional)

Apasswordtoverifyagainstthehash.

Returnstrueifthehashmatchesthegivenpassword.Returnsfalseotherwise.

Authentication

281

OAuth1.0aconstcreateOAuth1Client=require('@arangodb/foxx/oauth1');

TheOAuth1moduleprovidesabstractionsoverOAuth1.0aproviderslikeTwitter,XINGandTumblr.

Examples

Thefollowingextendstheusermanagementexample:


constoauth1=createOAuth1Client({

//We'lluseTwitterforthisexample

requestTokenEndpoint:'https://api.twitter.com/oauth/request_token',

authEndpoint:'https://api.twitter.com/oauth/authorize',

accessTokenEndpoint:'https://api.twitter.com/oauth/access_token',

activeUserEndpoint:'https://api.twitter.com/1.1/account/verify_credentials.json',

clientId:'keyboardcat',

clientSecret:'keyboardcat'

});

module.context.use('/oauth1',router);

//Seetheusermanagementexampleforsettingupthe

//sessionsandusersobjectsusedinthisexample

router.use(sessions);

router.post('/auth',function(req,res){

consturl=req.reverse('oauth1_callback');

constoauth_callback=req.makeAbsolute(url);

constrequestToken=oauth1.fetchRequestToken(oauth_callback);

if(requestToken.oauth_callback_confirmed!=='true'){

res.throw(500,'CouldnotfetchOAuthrequesttoken');

}

//Setrequesttokencookieforfiveminutes

res.cookie('oauth1_request_token',requestToken.oauth_token,{ttl:60*5});

//Redirecttotheprovider'sauthorizationURL

res.redirect(303,oauth1.getAuthUrl(requestToken.oauth_token));

});

router.get('/auth',function(req,res){

//MakesureCSRFcookiematchestheURL

constexpectedToken=req.cookie('oauth1_request_token');

if(!expectedToken||req.queryParams.oauth_token!==expectedToken){

res.throw(400,'CSRFmismatch.');

}

constauthData=oauth1.exchangeRequestToken(

req.queryParams.oauth_token,

req.queryParams.oauth_verifier

);

consttwitterToken=authData.oauth_token;

consttwitterSecret=authData.oauth_token_secret;

//Fetchtheactiveuser'sprofileinfo

constprofile=oauth1.fetchActiveUser(twitterToken,twitterSecret);

consttwitterId=profile.screen_name;

//TrytofindanexistinguserwiththeuserID

//(thisrequirestheuserscollection)

letuser=users.firstExample({twitterId});

if(user){

//UpdatethetwitterTokenifithaschanged

if(

user.twitterToken!==twitterToken||

user.twitterSecret!==twitterSecret

){

users.update(user,{twitterToken,twitterSecret});

}

}else{

//Createanewuserdocument

user={

username:`twitter:${twitterId}`,

twitterId,

twitterToken

OAuth1.0a

282

}



}

//Logtheuserin(thisrequiresthesessionmiddleware)


req.session.twitterToken=authData.twitterToken;

req.session.twitterSecret=authData.twitterSecret;


//Redirecttothedefaultroute

res.redirect(303,req.makeAbsolute('/'));

},'oauth1_callback')

.queryParam('oauth_token',joi.string().optional())

.queryParam('oauth_verifier',joi.string().optional());

CreatinganOAuth1.0aclientcreateOAuth1Client(options):OAuth1Client

CreatesanOAuth1.0aclient.

Arguments

options:Object


requestTokenEndpoint:string

Thefully-qualifiedURLoftheprovider'sTemporaryCredentialsRequestendpoint.ThisURLisusedtofetchtheunauthenticatedtemporarycredentialsthatwillbeusedtogeneratetheauthorizationredirectfortheuser.

authEndpoint:string

Thefully-qualifiedURLoftheprovider'sResourceOwnerAuthorizationendpoint.ThisistheURLtheuserwillberedirectedtoinordertoauthorizetheOAuthconsumer(i.e.yourservice).

accessTokenEndpoint:string

Thefully-qualifiedURLoftheprovider'sTokenRequestendpoint.ThisURLisusedtoexchangetheauthenticatedtemporarycredentialsreceivedfromtheauthorizationredirectfortheactualtokencredentialsthatcanbeusedtomakerequeststotheAPIserver.

activeUserEndpoint:string(optional)

Thefully-qualifiedURLoftheprovider'sendpointforfetchingdetailsaboutthecurrentuser.

clientId:string

Theapplication'sClientID(orConsumerKey)fortheprovider.

clientSecret:string

Theapplication'sClientSecret(orConsumerSecret)fortheprovider.

signatureMethod:string(Default:"HMAC-SHA1")

ThecryptographicmethodthatwillbeusedtosignOAuth1.0arequests.Only"HMAC-SHA1-"and"PLAINTEXT"aresupportedatthistime.

Notethatmanyprovidersmaynotimplement"PLAINTEXT"asitexposestheClientSecretandoauth_token_secretinsteadofgeneratingasignature.

ReturnsanOAuth1.0aclientforthegivenprovider.

SettingupOAuth1.0aforTwitter

IfyouwanttouseTwitterastheOAuth1.0aprovider,usethefollowingoptions:

OAuth1.0a

283

https://tools.ietf.org/html/rfc5849#section-2.1



requestTokenEndpoint:https://api.twitter.com/oauth/request_tokenauthEndpoint:https://api.twitter.com/oauth/authorizeaccessTokenEndpoint:https://api.twitter.com/oauth/access_tokenactiveUserEndpoint:https://api.twitter.com/1.1/account/verify_credentials.json

YoualsoneedtoobtainaclientIDandclientsecretfromTwitter:

1. CreatearegularaccountatTwitteroruseanexistingaccountyouown.2. VisittheTwitterApplicationManagementdashboardandsigninwithyourTwitteraccount.3. ClickonCreateNewAppandfollowtheinstructionsprovided.TheCallbackURLshouldmatchyouroauth_callbacklater.You

maybepromptedtoaddamobilephonenumbertoyouraccountandverifyit.4. OpentheKeysandAccessTonestab,thennotedowntheConsumerKeyandConsumerSecret.5. SettheoptionclientIdtotheConsumerKeyandtheoptionclientSecrettotheConsumerSecret.

Notethatifyouonlyneedread-onlyaccesstopublicinformation,youcanalsousetheclientIdandclientSecretdirectlywithoutOAuth1.0a.

SeeTwitterRESTAPIReferenceDocumentation.

SettingupOAuth1.0aforXING

IfyouwanttouseXINGastheOAuth1.0aprovider,usethefollowingoptions:

requestTokenEndpoint:https://api.xing.com/v1/request_tokenauthEndpoint:https://api.xing.com/v1/authorizeaccessTokenEndpoint:https://api.xing.com/v1/access_tokenactiveUserEndpoint:https://api.xing.com/v1/users/me

YoualsoneedtoobtainaclientIDandclientsecretfromXING:

1. CreatearegularaccountatXINGoruseanexistingaccountyouown.2. VisittheXINGDeveloperpageandsigninwithyourXINGaccount.3. ClickonCreateappandnotedowntheConsumerkeyandConsumersecret.4. SettheoptionclientIdtotheConsumerkeyandtheoptionclientSecrettotheConsumersecret.

SeeXINGDeveloperDocumentation.

SettingupOAuth1.0aforTumblr

IfyouwanttouseTumblrastheOAuth1.0aprovider,usethefollowingoptions:

requestTokenEndpoint:https://www.tumblr.com/oauth/request_tokenauthEndpoint:https://www.tumblr.com/oauth/authorizeaccessTokenEndpoint:https://www.tumblr.com/oauth/access_tokenactiveUserEndpoint:https://api.tumblr.com/v2/user/info

YoualsoneedtoobtainaclientIDandclientsecretfromTumblr:

1. CreatearegularaccountatTumblroruseanexistingaccountyouown.2. VisittheTumblrApplicationsdashboard.3. ClickonRegisterapplication,thenfollowtheinstructionsprovided.TheDefaultcallbackURLshouldmatchyouroauth_callback

later.4. NotedowntheOAuthConsumerKeyandSecretKey.Thesecretmaybehiddenbydefault.5. SettheoptionclientIdtotheOAuthConsumerKeyandtheoptionclientSecrettotheSecretKey.

SeeTumblrAPIDocumentation.

Fetchanunauthenticatedrequesttokenoauth1.fetchRequestToken(oauth_callback,opts)

Fetchesanoauth_tokenthatcanbeusedtocreateanauthorizationURLthatredirectstothegivenoauth_callbackonconfirmation.

OAuth1.0a

284

https://www.twitter.com

https://apps.twitter.com

https://dev.twitter.com/oauth/application-only

https://dev.twitter.com/rest/reference

https://xing.com

https://dev.xing.com

https://dev.xing.com/docs

https://www.tumblr.com

https://www.tumblr.com/oauth/apps

https://www.tumblr.com/docs/en/api/v2

PerformsaPOSTresponsetotherequestTokenEndpoint.

Throwsanexceptioniftheremoteserverrespondswithanemptyresponsebody.

Arguments

oauth_callback:string

Thefully-qualifiedURLofyourapplication'sOAuth1.0acallback.

opts:Object(optional)

Anobjectwithadditionalqueryparameterstoincludeintherequest.

SeeRFC5849.

Returnstheparsedresponseobject.

GettheauthorizationURLoauth1.getAuthUrl(oauth_token,opts):string

GeneratestheauthorizationURLfortheauthorizationendpoint.

Arguments

oauth_token:string

Theoauth_tokenpreviouslyreturnedbyfetchRequestToken.

opts:(optional)

AnobjectwithadditionalqueryparameterstoaddtotheURL.

SeeRFC5849.

Returnsafully-qualifiedURLfortheauthorizationendpointoftheproviderbyappendingtheoauth_tokenandanyadditionalargumentsfromoptstotheauthEndpoint.

Examples

constrequestToken=oauth1.fetchRequestToken(oauth_callback);

if(requestToken.oauth_callback_confirmed!=='true'){

thrownewError('ProvidercouldnotconfirmOAuth1.0callback');

}

constauthUrl=oauth1.getAuthUrl(requestToken.oauth_token);

Exchangeanauthenticatedrequesttokenforanaccesstokenoauth1.exchangeRequestToken(oauth_token,oauth_verifier,opts)

TakesapairofauthenticatedtemporarycredentialspassedtothecallbackURLbytheproviderandexchangesitforanoauth_tokenandoauth_token_secretthancanbeusedtoperformauthenticatedrequeststotheOAuth1.0aprovider.

PerformsaPOSTresponsetotheaccessTokenEndpoint.


Arguments

oauth_token:string

Theoauth_tokenpassedtothecallbackURLbytheprovider.

oauth_verifier:string

Theoauth_verifierpassedtothecallbackURLbytheprovider.


OAuth1.0a

285

https://tools.ietf.org/html/rfc5849



SeeRFC5849.


Fetchtheactiveuseroauth1.fetchActiveUser(oauth_token,oauth_token_secret,opts):Object

Fetchesdetailsoftheactiveuser.

PerformsaGETresponsetotheactiveUserEndpoint.


ReturnsnulliftheactiveUserEndpointisnotconfigured.

Arguments

oauth_token:string

AnOAuth1.0aaccesstokenasreturnedbyexchangeRequestToken.

oauth_token_secret:string

AnOAuth1.0aaccesstokensecretasreturnedbyexchangeRequestToken.



SeeRFC5849.


Examples

constauthData=oauth1.exchangeRequestToken(oauth_token,oauth_verifier);

constuserData=oauth1.fetchActiveUser(authData.oauth_token,authData.oauth_token_secret);

Createanauthenticatedrequestobjectoauth1.createSignedRequest(method,url,parameters,oauth_token,oauth_token_secret)

CreatesarequestobjectthatcanbeusedtoperformarequesttotheOAuth1.0aproviderwiththeprovidedtokencredentials.

Arguments

method:string

HTTPmethodtherequestwilluse,e.g."POST".

url:string

Thefully-qualifiedURLoftheprovidertherequestwillbeperformedagainst.

TheURLmayoptionallycontainanynumberofqueryparameters.

parameters:string|Object|null

Anadditionalobjectorquerystringcontainingqueryparametersorbodyparametersthatwillbepartofthesignedrequest.

oauth_token:string

AnOAuth1.0aaccesstokenasreturnedbyexchangeRequestToken.

oauth_token_secret:string

OAuth1.0a

286



AnOAuth1.0aaccesstokensecretasreturnedbyexchangeRequestToken.

Returnsanobjectwiththreeproperties:

url:ThenormalizedURLwithoutanyqueryparameters.

qs:Anormalizedquerystringcontainingallparametersandqueryparameters.

headers:Anobjectcontainingthefollowingproperties:

accept:Thestring"application/json".

authorization:AnOAuthauthorizationheadercontainingallOAuthparametersandtherequestsignature.

Examples

Fetchalistoftweetsmentioning@arangodb:

constrequest=require('@arangodb/request');

constreq=oauth1.createSignedRequest(

'GET',

'https://api.twitter.com/1.1/search/tweets.json',

{q:'@arangodb'},

authData.oauth_token,

authData.oauth_token_secret

);

constres=request(req);

console.log(res.json.statuses);

Signingamorecomplexrequest:

consturl='https://api.example.com/v1/timeline?visible=public';

constparams={hello:'world',longcat:'islong'};

constreq=oauth1.createSignedRequest(

'POST',

url,//URLincludesaqueryparameterthatwillbesigned

params,//Requestbodyneedstobesignedtoo

authData.oauth_token,

authData.oauth_token_secret

);

constres=request.post(url,{

form:params,

headers:{

accept:'application/x-www-form-urlencoded',

//Authorizationheaderincludesthesignature

authorization:req.headers.authorization

}

});

console.log(res.json);

OAuth1.0a

287

OAuth2.0constcreateOAuth2Client=require('@arangodb/foxx/oauth2');

TheOAuth2moduleprovidesabstractionsoverOAuth2.0providerslikeFacebook,GitHubandGoogle.

Examples

Thefollowingextendstheusermanagementexample:

constcrypto=require('@arangodb/crypto');


constoauth2=createOAuth2Client({

//We'lluseFacebookforthisexample

authEndpoint:'https://www.facebook.com/dialog/oauth',

tokenEndpoint:'https://graph.facebook.com/oauth/access_token',

activeUserEndpoint:'https://graph.facebook.com/v2.0/me',

clientId:'keyboardcat',

clientSecret:'keyboardcat'

});

module.context.use('/oauth2',router);

//Seetheusermanagementexampleforsettingupthe

//sessionsandusersobjectsusedinthisexample

router.use(sessions);

router.post('/auth',function(req,res){

constcsrfToken=crypto.genRandomAlphaNumbers(32);

consturl=req.reverse('oauth2_callback',{csrfToken});

constredirect_uri=req.makeAbsolute(url);

//SetCSRFcookieforfiveminutes

res.cookie('oauth2_csrf_token',csrfToken,{ttl:60*5});

//Redirecttotheprovider'sauthorizationURL

res.redirect(303,oauth2.getAuthUrl(redirect_uri));

});

router.get('/auth',function(req,res){

//Someproviderspasserrorsasqueryparameter

if(req.queryParams.error){

res.throw(500,`Providererror:${req.queryParams.error}`)

}

//MakesureCSRFcookiematchestheURL

constexpectedToken=req.cookie('oauth2_csrf_token');

if(!expectedToken||req.queryParams.csrfToken!==expectedToken){

res.throw(400,'CSRFmismatch.');

}

//MakesuretheURLcontainsagranttoken

if(!req.queryParams.code){

res.throw(400,'Providerdidnotpassgranttoken.');

}

//Reconstructtheredirect_uriusedforthegranttoken

consturl=req.reverse('oauth2_callback');

constredirect_uri=req.makeAbsolute(url);

//Fetchanaccesstokenfromtheprovider

constauthData=oauth2.exchangeGrantToken(

req.queryParams.code,

redirect_uri

);

constfacebookToken=authData.access_token;

//Fetchtheactiveuser'sprofileinfo

constprofile=oauth2.fetchActiveUser(facebookToken);

constfacebookId=profile.id;

//TrytofindanexistinguserwiththeuserID

//(thisrequirestheuserscollection)

letuser=users.firstExample({facebookId});

if(user){

//UpdatethefacebookTokenifithaschanged

if(user.facebookToken!==facebookToken){

users.update(user,{facebookToken});

}

}else{

OAuth2.0

288

//Createanewuserdocument

user={

username:`fb:${facebookId}`,

facebookId,

facebookToken

}



}

//Logtheuserin(thisrequiresthesessionmiddleware)


req.session.facebookToken=authData.facebookToken;


//Redirecttothedefaultroute

res.redirect(303,req.makeAbsolute('/'));

},'oauth2_callback')

.queryParam('error',joi.string().optional())

.queryParam('csrfToken',joi.string().optional())

.queryParam('code',joi.string().optional());

CreatinganOAuth2.0clientcreateOAuth2Client(options):OAuth2Client

CreatesanOAuth2.0client.

Arguments

options:Object


authEndpoint:string

Thefully-qualifiedURLoftheprovider'sauthorizationendpoint.

tokenEndpoint:string

Thefully-qualifiedURLoftheprovider'stokenendpoint.

refreshEndpoint:string(optional)

Thefully-qualifiedURLoftheprovider'srefreshtokenendpoint.

activeUserEndpoint:string(optional)

Thefully-qualifiedURLoftheprovider'sendpointforfetchingdetailsaboutthecurrentuser.

clientId:string

Theapplication'sClientID(orAppID)fortheprovider.

clientSecret:string

Theapplication'sClientSecret(orAppSecret)fortheprovider.

ReturnsanOAuth2.0clientforthegivenprovider.

SettingupOAuth2.0forFacebook

IfyouwanttouseFacebookastheOAuth2.0provider,usethefollowingoptions:

authEndpoint:https://www.facebook.com/dialog/oauthtokenEndpoint:https://graph.facebook.com/oauth/access_tokenactiveUserEndpoint:https://graph.facebook.com/v2.0/me

YoualsoneedtoobtainaclientIDandclientsecretfromFacebook:

1. CreatearegularaccountatFacebookoruseanexistingaccountyouown.2. VisittheFacebookDeveloperspage.

OAuth2.0

289

http://tools.ietf.org/html/rfc6749#section-3.1

http://tools.ietf.org/html/rfc6749#section-3.2

http://tools.ietf.org/html/rfc6749#section-6

https://www.facebook.com

https://developers.facebook.com

3. ClickonAppsinthemenu,thenselectRegisterasaDeveloper(theonlyoption)andfollowtheinstructionsprovided.Youmayneedtoverifyyouraccountbyphone.

4. ClickonAppsinthemenu,thenselectCreateaNewAppandfollowtheinstructionsprovided.5. Opentheappdashboard,thennotedowntheAppIDandAppSecret.Thesecretmaybehiddenbydefault.6. ClickonSettings,thenAdvancedandenteroneormoreValidOAuthredirectURIs.Atleastoneofthemmustmatchyourredirect_urilater.Don'tforgettosaveyourchanges.

7. SettheoptionclientIdtotheAppIDandtheoptionclientSecrettotheAppSecret.

SettingupOAuth2.0forGitHub

IfyouwanttouseGitHubastheOAuth2.0provider,usethefollowingoptions:

authEndpoint:https://github.com/login/oauth/authorize?scope=usertokenEndpoint:https://github.com/login/oauth/access_tokenactiveUserEndpoint:https://api.github.com/user

YoualsoneedtoobtainaclientIDandclientsecretfromGitHub:

1. CreatearegularaccountatGitHuboruseanexistingaccountyouown.2. GotoAccountSettings>Applications>Registernewapplication.3. ProvideanauthorizationcallbackURL.Thismustmatchyourredirect_urilater.4. Fillintheotherrequireddetailsandfollowtheinstructionsprovided.5. Opentheapplicationpage,thennotedowntheClientIDandClientSecret.6. SettheoptionclientIdtotheClientIDandtheoptionclientSecrettotheClientSecret.

SettingupOAuth2.0forGoogle

IfyouwanttouseGoogleastheOAuth2.0provider,usethefollowingoptions:

authEndpoint:https://accounts.google.com/o/oauth2/auth?access_type=offline&scope=profiletokenEndpoint:https://accounts.google.com/o/oauth2/tokenactiveUserEndpoint:https://www.googleapis.com/plus/v1/people/me

YoualsoneedtoobtainaclientIDandclientsecretfromGoogle:

1. CreatearegularaccountatGoogleoruseanexistingaccountyouown.2. VisittheGoogleDevelopersConsole.3. ClickonCreateProject,thenfollowtheinstructionsprovided.4. Whenyourprojectisready,opentheprojectdashboard,thenclickonEnableanAPI.5. EnabletheGoogle+APItoallowyourapptodistinguishbetweendifferentusers.6. OpentheCredentialspageandclickCreatenewClientID,thenfollowtheinstructionsprovided.AtleastoneAuthorizedRedirectURImustmatchyourredirect_urilater.AtleastoneAuthorizedJavaScriptOriginmustmatchyourapp'sfully-qualifieddomain.

7. WhentheClientIDisready,notedowntheClientIDandClientsecret.8. SettheoptionclientIdtotheClientIDandtheoptionclientSecrettotheClientsecret.

GettheauthorizationURLoauth2.getAuthUrl(redirect_uri,args):string

GeneratestheauthorizationURLfortheauthorizationendpoint.

Arguments

redirect_uri:string

Thefully-qualifiedURLofyourapplication'sOAuth2.0callback.

args:(optional)


response_type:string(Default:"code")

OAuth2.0

290

https://github.com

https://github.com/settings/applications/new

https://www.google.com

https://console.developers.google.com

SeeRFC6749.

Returnsafully-qualifiedURLfortheauthorizationendpointoftheproviderbyappendingtheclientIDandanyadditionalargumentsfromargstotheauthEndpoint.

Exchangeagrantcodeforanaccesstokenoauth2.exchangeGrantToken(code,redirect_uri)

Exchangesagrantcodeforanaccesstoken.

PerformsaPOSTresponsetothetokenEndpoint.


Arguments

code:string

Agrantcodereturnedbytheprovider'sauthorizationendpoint.

redirect_uri:string

TheoriginalcallbackURLwithwhichthecodewasrequested.

args:Object(optional)


grant_type:string(Default:"authorization_code")

SeeRFC6749.


Fetchtheactiveuseroauth2.fetchActiveUser(access_token):Object

Fetchesdetailsoftheactiveuser.

PerformsaGETresponsetotheactiveUserEndpoint.


ReturnsnulliftheactiveUserEndpointisnotconfigured.

Arguments

access_token:string

AnOAuth2.0accesstokenasreturnedbyexchangeGrantToken.


Examples

constauthData=oauth2.exchangeGrantToken(code,redirect_uri);

constuserData=oauth2.fetchActiveUser(authData.access_token);

OAuth2.0

291

http://tools.ietf.org/html/rfc6749

http://tools.ietf.org/html/rfc6749

TransactionsStartingwithversion1.3,ArangoDBprovidessupportforuser-definabletransactions.

TransactionsinArangoDBareatomic,consistent,isolated,anddurable(ACID).

TheseACIDpropertiesprovidethefollowingguarantees:

Theatomicityprinciplemakestransactionseithercompleteintheirentiretyorhavenoeffectatall.Theconsistencyprincipleensuresthatnoconstraintsorotherinvariantswillbeviolatedduringorafteranytransaction.Theisolationpropertywillhidethemodificationsofatransactionfromothertransactionsuntilthetransactioncommits.Finally,thedurabilitypropositionmakessurethatoperationsfromtransactionsthathavecommittedwillbemadepersistent.TheamountoftransactiondurabilityisconfigurableinArangoDB,asisthedurabilityoncollectionlevel.

Transactions

292

TransactioninvocationArangoDBtransactionsaredifferentfromtransactionsinSQL.

InSQL,transactionsarestartedwithexplicitBEGINorSTARTTRANSACTIONcommand.Followinganyseriesofdataretrievalormodificationoperations,anSQLtransactionisfinishedwithaCOMMITcommand,orrolledbackwithaROLLBACKcommand.Theremaybeclient/servercommunicationbetweenthestartandthecommit/rollbackofanSQLtransaction.

InArangoDB,atransactionisalwaysaserver-sideoperation,andisexecutedontheserverinonego,withoutanyclientinteraction.Alloperationstobeexecutedinsideatransactionneedtobeknownbytheserverwhenthetransactionisstarted.

TherearenoindividualBEGIN,COMMITorROLLBACKtransactioncommandsinArangoDB.Instead,atransactioninArangoDBisstartedbyprovidingadescriptionofthetransactiontothedb._executeTransactionJavaScriptfunction:

db._executeTransaction(description);

Thisfunctionwillthenautomaticallystartatransaction,executeallrequireddataretrievaland/ormodificationoperations,andattheendautomaticallycommitthetransaction.Ifanerroroccursduringtransactionexecution,thetransactionisautomaticallyaborted,andallchangesarerolledback.

Executetransaction

executesatransactiondb._executeTransaction(object)

Executesaserver-sidetransaction,asspecifiedbyobject.

objectmusthavethefollowingattributes:

collections:asub-objectthatdefineswhichcollectionswillbeusedinthetransaction.collectionscanhavetheseattributes:read:asinglecollectionoralistofcollectionsthatwillbeusedinthetransactioninread-onlymodewrite:asinglecollectionoralistofcollectionsthatwillbeusedinthetransactioninwriteorreadmode.

action:aJavascriptfunctionorastringwithJavascriptcodecontainingalltheinstructionstobeexecutedinsidethetransaction.Ifthecoderunsthroughsuccessfully,thetransactionwillbecommittedattheend.Ifthecodethrowsanexception,thetransactionwillberolledbackandalldatabaseoperationswillberolledback.

Additionally,objectcanhavethefollowingoptionalattributes:

waitForSync:booleanflagindicatingwhetherthetransactionisforcedtobesynchronous.lockTimeout:anumericvaluethatcanbeusedtosetatimeoutforwaitingoncollectionlocks.Ifnotspecified,adefaultvaluewillbeused.SettinglockTimeoutto0willmakeArangoDBnottimeoutwaitingforalock.params:optionalargumentspassedtothefunctionspecifiedinaction.

ThefollowingattributescanbeusedfortransactionsintheRocksDBstorageengine:

maxTransactionSize:transactionsizelimitinbytesintermediateCommitSize:maximumtotalsizeofoperationsafterwhichanintermediatecommitisperformedautomaticallyintermediateCommitCount:maximumnumberofoperationsafterwhichanintermediatecommitisperformedautomatically

Declarationofcollections

Allcollectionswhicharetoparticipateinatransactionneedtobedeclaredbeforehand.Thisisanecessitytoensureproperlockingandisolation.

Collectionscanbeusedinatransactioninwritemodeorinread-onlymode.

Ifanydatamodificationoperationsaretobeexecuted,thecollectionmustbedeclaredforuseinwritemode.Thewritemodeallowsmodifyingandreadingdatafromthecollectionduringthetransaction(i.e.thewritemodeincludesthereadmode).

Contrary,usingacollectioninread-onlymodewillonlyallowperformingreadoperationsonacollection.Anyattempttowriteintoacollectionusedinread-onlymodewillmakethetransactionfail.


293

Collectionsforatransactionaredeclaredbyprovidingtheminthecollectionsattributeoftheobjectpassedtothe_executeTransactionfunction.Thecollectionsattributehasthesub-attributesreadandwrite:

db._executeTransaction({

collections:{

write:["users","logins"],

read:["recommendations"]

}

});

readandwriteareoptionalattributes,andonlyneedtobespecifiediftheoperationsinsidethetransactionsdemandforit.

Thecontentsofreadorwritecaneachbelistsarrayscollectionnamesorasinglecollectionname(asastring):


collections:{

write:"users",

read:"recommendations"

}

});

Note:Itiscurrentlyoptionaltospecifycollectionsforread-onlyaccess.Evenwithoutspecifyingthem,itisstillpossibletoreadfromsuchcollectionsfromwithinatransaction,butwithrelaxedisolation.PleaserefertoTransactionsLockingformoredetails.

Inordertomakeatransactionfailwhenanon-declaredcollectionisusedinsideforreading,theoptionalallowImplicitsub-attributeofcollectionscanbesettofalse:


collections:{

read:"recommendations",

allowImplicit:false/*thisdisallowsreadaccesstoothercollections

thanspecified*/

},

action:function(){

vardb=require("@arangodb").db;

returndb.foobar.toArray();/*willfailbecausedb.foobarmustnotbeaccessed

forreadinginsidethistransaction*/

}

});

ThedefaultvalueforallowImplicitistrue.Write-accessingcollectionsthathavenotbeendeclaredinthecollectionsarrayisneverpossible,regardlessofthevalueofallowImplicit.

Declarationofdatamodificationandretrievaloperations

AlldatamodificationandretrievaloperationsthataretobeexecutedinsidethetransactionneedtobespecifiedinaJavascriptfunction,usingtheactionattribute:


collections:{

write:"users"

},

action:function(){

//alloperationsgohere

}

});

AnyvalidJavascriptcodeisallowedinsideactionbutthecodemayonlyaccessthecollectionsdeclaredincollections.actionmaybeaJavascriptfunctionasshownabove,orastringrepresentationofaJavascriptfunction:


collections:{

write:"users"

},

action:"function(){doSomething();}"

});


294

Pleasenotethatanyoperationsspecifiedinactionwillbeexecutedontheserver,inaseparatescope.Variableswillbeboundlate.AccessinganyJavaScriptvariablesdefinedontheclient-sideorinsomeotherservercontextfrominsideatransactionmaynotwork.Instead,anyvariablesusedinsideactionshouldbedefinedinsideactionitself:


collections:{

write:"users"

},

action:function(){

vardb=require(...).db;

db.users.save({...});

}

});

Whenthecodeinsidetheactionattributeisexecuted,thetransactionisalreadystartedandallrequiredlockshavebeenacquired.Whenthecodeinsidetheactionattributefinishes,thetransactionwillautomaticallycommit.Thereisnoexplicitcommitcommand.

Tomakeatransactionabortandrollbackallchanges,anexceptionneedstobethrownandnotcaughtinsidethetransaction:


collections:{

write:"users"

},

action:function(){


db.users.save({_key:"hello"});

//willabortandrollbackthetransaction

throw"doh!";

}

});

Thereisnoexplicitabortorrollbackcommand.

Asmentionedearlier,atransactionwillcommitautomaticallywhentheendoftheactionfunctionisreachedandnoexceptionhasbeenthrown.Inthiscase,theusercanreturnanylegalJavaScriptvaluefromthefunction:


collections:{

write:"users"

},

action:function(){


db.users.save({_key:"hello"});

//willcommitthetransactionandreturnthevalue"hello"

return"hello";

}

});

Customexceptions

Onemaywishtodefinecustomexceptionsinsideofatransaction.Tohavetheexceptionpropagateupwardsproperly,pleasethrowananinstanceofbaseJavaScriptErrorclassoraderivative.Tospecifyanerrornumber,includeitastheerrorNumberfield.Asanexample:


collections:{},

action:function(){

varerr=newError('Myerrorcontext');

err.errorNumber=1234;

throwerr;

}

});


295

Note:Inpreviousversions,customexceptionswhichdidnothaveanError-likeformweresimplyconvertedtostringsandexposedintheexceptionfieldofthereturnederror.Thisisnolongerthecase,asithadthepotentialtoleakunwantedinformationifimproperlyused.

Examples

Thefirstexamplewillwrite3documentsintoacollectionnamedc1.Thec1collectionneedstobedeclaredinthewriteattributeofthecollectionsattributepassedtotheexecuteTransactionfunction.

Theactionattributecontainstheactualtransactioncodetobeexecuted.Thiscodecontainsalldatamodificationoperations(3inthisexample).

//setup

db._create("c1");


collections:{

write:["c1"]

},

action:function(){


db.c1.save({_key:"key1"});



}

});

db.c1.count();//3

Abortingthetransactionbythrowinganexceptionintheactionfunctionwillrevertallchanges,soasifthetransactionneverhappened:

//setup

db._create("c1");


collections:{

write:["c1"]

},

action:function(){



db.c1.count();//1


db.c1.count();//2

throw"doh!";

}

});

db.c1.count();//0

Theautomaticrollbackisalsoexecutedwhenaninternalexceptionisthrownatsomepointduringtransactionexecution:

//setup

db._create("c1");


collections:{

write:["c1"]

},

action:function(){



//willthrowduplicateakeyerror,notexplicitlyrequestedbytheuser


//we'llnevergethere...

}

});

db.c1.count();//0


296

Asrequiredbytheconsistencyprinciple,abortingorrollingbackatransactionwillalsorestoresecondaryindexestothestateattransactionstart.

Cross-collectiontransactions

There'salsothepossibilitytorunatransactionacrossmultiplecollections.Inthiscase,multiplecollectionsneedtobedeclaredinthecollectionsattribute,e.g.:

//setup

db._create("c1");

db._create("c2");


collections:{

write:["c1","c2"]

},

action:function(){




}

});

db.c1.count();//1

db.c2.count();//1

Again,throwinganexceptionfrominsidetheactionfunctionwillmakethetransactionabortandrollbackallchangesinallcollections:

//setup

db._create("c1");

db._create("c2");


collections:{

write:["c1","c2"]

},

action:function(){


for(vari=0;i<100;++i){

db.c1.save({_key:"key"+i});

db.c2.save({_key:"key"+i});

}

db.c1.count();//100

db.c2.count();//100

//abort

throw"doh!"

}

});

db.c1.count();//0

db.c2.count();//0


297

PassingparameterstotransactionsArbitraryparameterscanbepassedtotransactionsbysettingtheparamsattributewhendeclaringthetransaction.Thisfeatureishandytore-usethesametransactioncodeformultiplecallsbutwithdifferentparameters.

Abasicexample:


collections:{},

action:function(params){

returnparams[1];

},

params:[1,2,3]

});

Theaboveexamplewillreturn2.

Someexamplethatusescollections:


collections:{

write:"users",

read:["c1","c2"]

},

action:function(params){

vardb=require('@arangodb').db;

vardoc=db.c1.document(params['c1Key']);

db.users.save(doc);

doc=db.c2.document(params['c2Key']);

db.users.save(doc);

},

params:{

c1Key:"foo",

c2Key:"bar"

}

});

Passingparameters

298

LockingandIsolationTransactionsneedtospecifyfromwhichcollectionstheywillreaddataandwhichcollectionstheyintenddomodify.Thiscanbedonebysettingtheread,write,orexclusiveattributesinthecollectionsattributeofthetransaction:


collections:{

read:"users",

write:["test","log"]

},

action:function(){

constdb=require("@arangodb").db;

db.users.toArray().forEach(function(doc){

db.log.insert({value:"removeduser:"+doc.name});

db.test.remove(doc._key);

});

}

});

writeheremeanswriteaccesstothecollection,andalsoincludesanyreadaccesses.exclusiveisasynonymforwriteintheMMFilesengine,becausebothexclusiveandwritewillacquirecollection-levellocksinthisengine.IntheRocksDBengine,exclusivemeansexclusivewriteaccesstothecollection,andwritemeans(shared)writeaccesstothecollection,whichcanbeinterleavedwithwriteaccessesbyotherconcurrenttransactions.

MMFilesengine

TheMMFilesengineusesthefollowinglockingmechanismstoserializetransactionsonthesamedata:

Allcollectionsspecifiedinthecollectionsattributearelockedintherequestedmode(readorwrite)attransactionstart.Lockingofmultiplecollectionsisperformedinalphabeticalorder.Whenatransactioncommitsorrollsback,alllocksarereleasedinreverseorder.Thelockingorderisdeterministictoavoiddeadlocks.

Whilelocksareheld,modificationsbyothertransactionstothecollectionsparticipatinginthetransactionareprevented.Atransactionwillthusseeaconsistentviewoftheparticipatingcollections'data.

Additionally,atransactionwillnotbeinterruptedorinterleavedwithanyotherongoingoperationsonthesamecollection.Thismeanseachtransactionwillruninisolation.Atransactionshouldneverseeuncommittedorrolledbackmodificationsbyothertransactions.Additionally,readsinsideatransactionarerepeatable.

Notethattheaboveistrueonlyforallcollectionsthataredeclaredinthecollectionsattributeofthetransaction.

RocksDBengine

TheRocksDBenginedoesnotlockanycollectionsparticipatinginatransactionforread.Readoperationscanruninparalleltootherreadorwriteoperationsonthesamecollections.

Forallcollectionsthatareusedinwritemode,theRocksDBenginewillinternallyacquirea(shared)readlock.Thismeansthatmanywriterscanmodifydatainthesamecollectioninparallel(andalsoruninparalleltoongoingreads).However,iftwoconcurrenttransactionsattempttomodifythesamedocumentorindexentry,therewillbeawrite-writeconflict,andoneofthetransactionswillabortwitherror1200("conflict").Itisthenuptoclientapplicationstoretrythefailedtransactionoracceptthefailure.

Inordertoguardlong-runningorcomplextransactionsagainstconcurrentoperationsonthesamedata,theRocksDBengineallowstoaccesscollectionsinexclusivemode.Exclusiveaccesseswillinternallyacquireawrite-lockonthecollections,sotheyarenotexecutedinparallelwithanyotherwriteoperations.Readoperationscanstillbecarriedoutbyotherconcurrenttransactions.

Lazilyaddingcollections

Theremightbesituationswhendeclaringallcollectionsaprioriisnotpossible,forexample,becausefurthercollectionsaredeterminedbyadynamicAQLqueryinsidethetransaction,forexampleaqueryusingAQLgraphtraversal.

Lockingandisolation

299

Inthiscase,itwouldbeimpossibletoknowbeforehandwhichcollectiontolock,andthusitislegaltonotdeclarecollectionsthatwillbeaccessedinthetransactioninread-onlymode.Accessinganon-declaredcollectioninread-onlymodeduringatransactionwilladdthecollectiontothetransactionlazily,andfetchdatafromthecollectionasusual.However,asthecollectionisaddedlazily,thereisnoisolationfromotherconcurrentoperationsortransactions.Readsfromsuchcollectionsarepotentiallynon-repeatable.

Examples:


collections:{

read:"users"

},

action:function(){


/*ExecuteanAQLquerythattraversesagraphstartingata"users"vertex.

Itisyetunknownintowhichothercollectionsthequerymighttraverse*/

db._createStatement({

query:`FORvINANY"users/1234"connectionsRETURNv`

}).execute().toArray().forEach(function(d){

/*...*/

});

}

});

Thisautomaticlazyadditionofcollectionstoatransactionalsointroducesthepossibilityofdeadlocks.Deadlocksmayoccurifthereareconcurrenttransactionsthattrytoacquirelocksonthesamecollectionslazily.

Inordertomakeatransactionfailwhenanon-declaredcollectionisusedinsideatransactionforreading,theoptionalallowImplicitsub-attributeofcollectionscanbesettofalse:


collections:{

read:"users",

allowImplicit:false

},

action:function(){

/*Thebelowquerywillnowfailbecausethecollection"connections"hasnot

beenspecifiedinthelistofcollectionsusedbythetransaction*/


db._createStatement({

query:`FORvINANY"users/1234"connectionsRETURNv`

}).execute().toArray().forEach(function(d){

/*...*/

});

}

});

ThedefaultvalueforallowImplicitistrue.Write-accessingcollectionsthathavenotbeendeclaredinthecollectionsarrayisneverpossible,regardlessofthevalueofallowImplicit.

Ifusers/1234hasanedgeinconnections,linkingittoanotherdocumentintheuserscollection,thenthefollowingexplicitdeclarationwillwork:


collections:{

read:["users","connections"],

allowImplicit:false

},

/*...*/

Iftheedgepointstoadocumentinanothercollectionhowever,thenthequerywillfail,unlessthatothercollectionisaddedtothedeclarationaswell.

Notethatifadocumenthandleisusedasstartingpointforatraversal,e.g.FORvINANY"users/not_linked"...orFORvINANY{_id:"users/not_linked"}...,thennoerrorisraisedinthecaseofthestartvertexnothavinganyedgestofollow,withallowImplicit:falseandusersnotbeingdeclaredforreadaccess.AQLonlyseesastringanddoesnotconsideritareadaccess,unlessthereareedgesconnectedtoit.FORvINANYDOCUMENT("users/not_linked")...willfailevenwithoutedges,asitisalwaysconsideredtobeareadaccesstotheuserscollection.

Lockingandisolation

300

DeadlocksandDeadlockdetection

Adeadlockisasituationinwhichtwoormoreconcurrentoperations(usertransactionsorAQLqueries)trytoaccessthesameresources(collections,documents)andneedtowaitfortheotherstofinish,butnoneofthemcanmakeanyprogress.

AgoodexampleforadeadlockistwoconcurrentlyexecutingtransactionsT1andT2thattrytoaccessthesamecollectionsbutthatneedtowaitforeachother.Inthisexample,transactionT1willwritetocollectionc1,butwillalsoreaddocumentsfromcollectionc2withoutannouncingit:


collections:{

write:"c1"

},

action:function(){


/*writeintoc1(announced)*/

db.c1.insert({foo:"bar"});

/*someoperationherethattakeslongtoexecute...*/

/*readfromc2(unannounced)*/

db.c2.toArray();

}

});

TransactionT2announcestowriteintocollectionc2,butwillalsoreaddocumentsfromcollectionc1withoutannouncingit:


collections:{

write:"c2"

},

action:function(){


/*writeintoc2(announced)*/

db.c2.insert({bar:"baz"});

/*someoperationherethattakeslongtoexecute...*/

/*readfromc1(unannounced)*/

db.c1.toArray();

}

});

Intheaboveexample,adeadlockwilloccuriftransactionT1andT2havebothacquiredtheirwritelocks(T1forcollectionc1andT2forcollectionc2)andarethentryingtoreadfromtheotherother(T1willreadfromc2,T2willreadfromc1).T1willthentrytoacquirethereadlockoncollectionc2,whichispreventedbytransactionT2.T2howeverwillwaitforthereadlockoncollectionc1,whichispreventedbytransactionT1.

Incaseofsuchdeadlock,therewouldbenoprogressforanyoftheinvolvedtransactions,andnoneoftheinvolvedtransactionscouldevercomplete.Thisiscompletelyundesirable,sotheautomaticdeadlockdetectionmechanisminArangoDBwillautomaticallyabortoneofthetransactionsinvolvedinsuchdeadlock.Abortingmeansthatallchangesdonebythetransactionwillberolledbackanderror29(deadlockdetected)willbethrown.

Clientcode(AQLqueries,usertransactions)thataccessesmorethanonecollectionshouldbeawareofthepotentialofdeadlocksandshouldhandletheerror29(deadlockdetected)properly,eitherbypassingtheexceptiontothecallerorretryingtheoperation.

Toavoidbothdeadlocksandnon-repeatablereads,allcollectionsusedinatransactionshouldbespecifiedinthecollectionsattributewhenknowninadvance.Incasethisisnotpossiblebecausecollectionsareaddeddynamicallyinsidethetransaction,deadlocksmayoccurandthedeadlockdetectionmaykickinandabortthetransaction.

TheRocksDBengineusesdocument-levellocksandthereforewillnothaveadeadlockproblemoncollectionlevel.Iftwoconcurrenttransactionshowevermodifythesamedocumentsorindexentries,theRocksDBenginewillsignalawrite-writeconflictandabortoneofthetransactionswitherror1200("conflict")automatically.

Lockingandisolation

301

Lockingandisolation

302

DurabilityTransactionsareexecutedinmainmemoryfirstuntilthereiseitherarollbackoracommit.Onrollback,nodatawillbewrittentodisk,buttheoperationsfromthetransactionwillbereversedinmemory.

Oncommit,allmodificationsdoneinthetransactionwillbewrittentothecollectiondatafiles.ThesewriteswillbesynchronizedtodiskifanyofthemodifiedcollectionshasthewaitForSyncpropertysettotrue,orifanyindividualoperationinthetransactionwasexecutedwiththewaitForSyncattribute.Additionally,transactionsthatmodifydatainmorethanonecollectionareautomaticallysynchronizedtodisk.Thissynchronizationisdonetonotonlyensuredurability,buttoalsoensureconsistencyincaseofaservercrash.

Thatmeansifyouonlymodifydatainasinglecollection,andthatcollectionhasitswaitForSyncpropertysettofalse,thewholetransactionwillnotbesynchronizedtodiskinstantly,butwithasmalldelay.

Thereisthusthepotentialriskoflosingdatabetweenthecommitofthetransactionandtheactual(delayed)disksynchronization.ThisisthesameaswritingintocollectionsthathavethewaitForSyncpropertysettofalseoutsideofatransaction.IncaseofacrashwithwaitForSyncsettofalse,theoperationsperformedinthetransactionwilleitherbevisiblecompletelyornotatall,dependingonwhetherthedelayedsynchronizationhadkickedinornot.

ToensuredurabilityoftransactionsonacollectionthathavethewaitForSyncpropertysettofalse,youcansetthewaitForSyncattributeoftheobjectthatispassedtoexecuteTransaction.ThiswillforceasynchronizationofthetransactiontodiskevenforcollectionsthathavewaitForSyncsettofalse:


collections:{

write:"users"

},

waitForSync:true,

action:function(){...}

});

Analternativeistoperformanoperationwithanexplicitsyncrequestinatransaction,e.g.

db.users.save({_key:"1234"},true);

Inthiscase,thetruevaluewillmakethewholetransactionbesynchronizedtodiskatthecommit.

Inanycase,ArangoDBwillgiveusersthechoiceofwhetherornottheywantfulldurabilityforsinglecollectiontransactions.Usingthedelayedsynchronization(i.e.waitForSyncwithavalueoffalse)willpotentiallyincreasethroughputandperformanceoftransactions,butwillintroducetheriskoflosingthelastcommittedtransactionsinthecaseofacrash.

Incontrast,transactionsthatmodifydatainmorethanonecollectionareautomaticallysynchronizedtodisk.Thiscomesatthecostofseveraldisksync.Foramulti-collectiontransaction,thecalltothe_executeTransactionfunctionwillonlyreturnafterthedataofallmodifiedcollectionshasbeensynchronizedtodiskandthetransactionhasbeenmadefullydurable.Thisnotonlyreducestheriskoflosingdataincaseofacrashbutalsoensuresconsistencyafterarestart.

Incaseofaservercrash,anymulti-collectiontransactionsthatwerenotyetcommittedorinpreparationtobecommittedwillberolledbackonserverrestart.

Formulti-collectiontransactions,therewillbeatleastonedisksyncoperationpermodifiedcollection.Multi-collectiontransactionsthushaveapotentiallyhighercostthansinglecollectiontransactions.Thereisnoconfigurationtoturnoffdisksynchronizationformulti-collectiontransactionsinArangoDB.Thedisksyncspeedofthesystemwillthusbethemostimportantfactorfortheperformanceofmulti-collectiontransactions.

Durability

303

Limitations

InGeneral

TransactionsinArangoDBhavebeendesignedwithparticularusecasesinmind.Theywillbemainlyusefulforshortandsmalldataretrievaland/ormodificationoperations.

Theimplementationisnotoptimizedforverylong-runningorveryvoluminousoperations,andmaynotbeusableforthesecases.

Onelimitationisthatatransactionoperationinformationmustfitintomainmemory.Thetransactioninformationconsistsofrecordpointers,revisionnumbersandrollbackinformation.Theactualdatamodificationoperationsofatransactionarewrittentothewrite-aheadloganddonotneedtofitentirelyintomainmemory.

Ongoingtransactionswillalsopreventthewrite-aheadlogsfrombeingfullygarbage-collected.Informationinthewrite-aheadlogfilescannotbewrittentocollectiondatafilesorbediscardedwhiletransactionsareongoing.

Toensureprogressofthewrite-aheadloggarbagecollection,transactionsshouldbekeptassmallaspossible,andbigtransactionsshouldbesplitintomultiplesmallertransactions.

TransactionsinArangoDBcannotbenested,i.e.atransactionmustnotstartanothertransaction.Ifanattemptismadetocallatransactionfrominsidearunningtransaction,theserverwillthrowerror1651(nestedtransactionsdetected).

ItisalsodisallowedtoexecuteusertransactiononsomeofArangoDB'sownsystemcollections.Thisshouldn'tbeaproblemforregularusageassystemcollectionswillnotcontainuserdataandthereisnoneedtoaccessthemfromwithinausertransaction.

Someoperationsarenotallowedinsidetransactionsingeneral:

creationanddeletionofdatabases(db._createDatabase(),db._dropDatabase())creationanddeletionofcollections(db._create(),db._drop(),db.<collection>.rename())creationanddeletionofindexes(db.<collection>.ensureIndex(),db.<collection>.dropIndex())

Ifanattemptismadetocarryoutanyoftheseoperationsduringatransaction,ArangoDBwillabortthetransactionwitherrorcode1653(disallowedoperationinsidetransaction).

Finally,allcollectionsthatmaybemodifiedduringatransactionmustbedeclaredbeforehand,i.e.usingthecollectionsattributeoftheobjectpassedtothe_executeTransactionfunction.Ifanyattemptismadetocarryoutadatamodificationoperationonacollectionthatwasnotdeclaredinthecollectionsattribute,thetransactionwillbeabortedandArangoDBwillthrowerror1652unregisteredcollectionusedintransaction.Itislegaltonotdeclareread-onlycollections,butthisshouldbeavoidedifpossibletoreducetheprobabilityofdeadlocksandnon-repeatablereads.

PleaserefertoLockingandIsolationformoredetails.

InClustersUsingasingleinstanceofArangoDB,multi-document/multi-collectionqueriesareguaranteedtobefullyACID.ThisismorethanmanyotherNoSQLdatabasesystemssupport.Inclustermode,single-documentoperationsarealsofullyACID.Multi-document/multi-collectionqueriesinaclusterarenotACID,whichisequallythecasewithcompetingdatabasesystems.TransactionsinaclusterwillbesupportedinafutureversionofArangoDBandmaketheseoperationsfullyACIDaswell.Notethatfornon-shardedcollectionsinacluster,thetransactionalpropertiesofasingleserverapply(fullyACID).

TransactionsintheRocksDBstorageengineDataofongoingtransactionsisstoredinRAM.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatacreatedormodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.

ThefollowingglobaloptionscanbeusedtocontroltheRAMusageandautomaticintermediatecommitsfortheRocksDBengine:

Limitations

304

--rocksdb.max-transaction-size

Transactionsizelimit(inbytes).TransactionsstoreallkeysandvaluesinRAM,solargetransactionsruntheriskofcausingout-of-memorysitations.Thissettingallowsyoutoensurethatdoesnothappenbylimitingthesizeofanyindividualtransaction.TransactionswhoseoperationswouldconsumemoreRAMthanthisthresholdvaluewillabortautomaticallywitherror32("resourcelimitexceeded").

--rocksdb.intermediate-commit-size

Ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.

--rocksdb.intermediate-commit-count

Ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.

Theabovevaluescanalsobeadjustedpertransaction,bysettingthefollowingattributesinthecalltodb._executeTransaction():

maxTransactionSize:transactionsizelimitinbytesintermediateCommitSize:maximumtotalsizeofoperationsafterwhichanintermediatecommitisperformedautomaticallyintermediateCommitCount:maximumnumberofoperationsafterwhichanintermediatecommitisperformedautomatically

Limitations

305

DeploymentInthischapterwedescribevariouspossibilitiestodeployArangoDB.Inparticularfortheclustermode,therearedifferentwaysandwewanttohighlighttheiradvantagesanddisadvantages.Weevendocumentindetail,howtosetupaclusterbysimplystartingvariousArangoDBprocessesondifferentmachines,eitherdirectlyorusingDockercontainers.

SingleinstanceCluster

DC/OS,ApacheMesosandMarathonGeneric&DockerAdvancedTopics

StandaloneAgencyTestsetuponalocalmachineStartingprocessesondifferentmachinesLaunchinganArangoDBclusterusingDockercontainers

MultipleDatacenters

Deployment

306

Singleinstancedeployment

ThelatestofficialbuildsofArangoDBforallsupportedoperatingsystemsmaybeobtainedfromhttps://www.arangodb.com/download/.

Linuxremarks

Besidestheofficialimageswhichareprovidedforthemostpopularlinuxdistributionstherearealsoavarietyofunofficialimagesprovidedbythecommunity.Wearetrackingmostofthecommunitycontributions(includingneworupdatedimages)inournewsletter:

https://www.arangodb.com/category/newsletter/

Windowsremarks

PleasenotethatArangoDBwillonlyworkon64bit.

Docker

ThesimplestwaytodeployArangoDBisusingDocker.TogetageneralunderstandingofDockerhavealookattheirexcellentdocumentation.

Authentication

TostarttheofficialDockercontaineryouwillhavetodecideonanauthenticationmethod.Otherwisethecontainerwon'tstart.

ProvideoneoftheargumentstoDockerasanenvironmentvariable.

Therearethreeoptions:

1. ARANGO_NO_AUTH=1

Disableauthenticationcompletely.Usefulforlocaltestingorforoperatinginatrustednetwork(withoutapublicinterface).

2. ARANGO_ROOT_PASSWORD=password

StartArangoDBwiththegivenpasswordforroot

3. ARANGO_RANDOM_ROOT_PASSWORD=1

LetArangoDBgeneratearandomrootpassword

Togetgoingquickly:

dockerrun-eARANGO_RANDOM_ROOT_PASSWORD=1arangodb/arangodb

ForanindepthguideaboutDockerandArangoDBpleasechecktheofficialdocumentation:https://hub.docker.com/r/arangodb/arangodb/.Notethatweareusingtheimagearangodb/arangodbherewhichisalwaysthemostcurrentone.Thereisalsothe"official"onecalledarangodbwhosedocumentationishere:https://hub.docker.com/_/arangodb/

Singleinstance

307

https://www.arangodb.com/download/

https://www.arangodb.com/category/newsletter/

https://docker.io/

https://docs.docker.com/

https://hub.docker.com/r/arangodb/arangodb/

https://hub.docker.com/_/arangodb/

ClusterMesos,DC/OS:DistributeddeploymentusingApacheMesos

Generic&Docker:AutomaticnativeclusterswithArangoDBStarter

AdvancedTopics:StandaloneAgency,local/distributed/Dockerclusters

Cluster

308

DistributeddeploymentusingApacheMesos

ArangoDBhasasophisticatedandyeteasywaytouseclustermode.Toleveragethefullclusterfeatureset(monitoring,scaling,automaticfailoverandautomaticreplacementoffailednodes)youhavetorunArangoDBonsomekindofclustermanagementsystem.CurrentlyArangoDBreliesonApacheMesosinthatmatter.Mesosisaclusteroperatingsystemwhichpowerssomeoftheworldsbiggestdatacentersrunningseveralthousandsofnodes.

DC/OS

DC/OSistherecommendedwaytoinstallaclusterasiteasesmuchoftheprocesstoinstallaMesoscluster.YoucandeployitveryquicklyonavarietyofcloudhostersorsetupyourownDC/OSlocally.DC/OSisasetoftoolsbuiltontopofApacheMesos.ApacheMesosisasocalled"DistributedClusterOperationSystem"andthecoreofDC/OS.ApacheMesoshastheconceptofsocalledpersistentvolumeswhichmakeitperfectlysuitableforadatabase.

Installing

FirstprepareaDC/OSclusterbygoingtohttps://dcos.ioandfollowingtheinstructionsthere.

DC/OScomeswithitsownpackagemanagement.Packagescanbeinstalledfromthesocalled"Universe".AsanofficialDC/OSpartnerArangoDBcanbeinstalledfromtherestraightaway.

1. InstallingviaDC/OSUI

i. OpenyourbrowserandgototheDC/OSadmininterfaceii. Openthe"Universe"tabiii. Locatearangodbandhit"InstallPackage"iv. Press"InstallPackage"

2. InstallingviatheDC/OScommandline

i. Installthedcoscliii. Openaterminalandissuedcosinstallarangodb

Bothoptionsareessentiallydoingthesameinthebackground.BotharestartingArangoDBwithitsdefaultoptionsset.

Toreviewthedefaultoptionsusingthewebinterfacesimplyclick"AdvancedInstallation"inthewebinterface.Thereyouwillfindalistofoptionsincludingsomeexplanation.

ToreviewthedefaultoptionsusingtheCLIfirsttypedcospackagedescribe--configarangodb.Thiswillgiveyouaflatlistofdefaultsettings.

Togetanexplanationofthevariouscommandlineoptionspleasecheckthelatestoptionshere(choosethemostrecentnumberandhavealookatconfig.json):

https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/A/arangodb

AfterinstallationDC/OSwillstartdeployingtheArangoDBclusterontheDC/OScluster.YoucanwatchArangoDBstartingonthe"Services"tabinthewebinterface.OnceitislistedashealthyclickthelinknexttoitandyoushouldseetheArangoDBwebinterface.

ArangoDBMesosframework

AssoonasArangoDBwasdeployedMesoswillkeepyourclusterrunning.ThewebinterfacehasmanymonitoringfacilitiessobesuretomakeyourselffamiliarwiththeDC/OSwebinterface.AsafaulttolerantsystemMesoswilltakecareofmostfailurescenariosautomatically.MesosdoesthatbyrunningArangoDBasasocalled"framework".ThisframeworkhasbeenspecificallybuilttokeepArangoDBrunninginahealthyconditionontheMesoscluster.Fromtimetotimeataskmightfail.TheArangoDBframeworkwillthentakecareofreschedulingthefailedtask.Asitknowsabouttheveryspecificsofeachclustertaskanditsroleitwillautomaticallytakecareofmostfailurescenarios.

Toinspectwhattheframeworkisdoinggotohttp://web-interface-url/mesosinyourbrowser.Locatethetask"arangodb"andinspectstderrinthe"Sandbox".Thiscanbeofinterestforexamplewhenaslavegotlostandtheframeworkisreschedulingthetask.

Mesos,DC/OS

309

http://mesos.apache.org/documentation/latest/persistent-volume/

https://dcos.io

https://docs.mesosphere.com/usage/cli/

https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/A/arangodb

UsingArangoDB

TouseArangoDBasadatastoreinyourDC/OSclusteryoucanfacilitatetheservicediscoveryofDC/OS.AssumingyoudeployedastandardArangoDBclusterthemesosdnswillknowaboutarangodb.mesos.BydoingaSRVDNSrequest(checkthedocumentationofmesosdns)youcanfindouttheportwheretheinternalHAProxyofArangoDBisrunning.ThiswillofferaroundrobinloadbalancertoaccessallArangoDBcoordinators.

ScalingArangoDB

TochangethesettingsofyourArangoDBClusteraccesstheArangoDBUIandhit"Nodes".Onthescaletabyouwillhavetheabilitytoscaleyourclusterupanddown.

AfterchangingthesettingstheArangoDBframeworkwilltakecareoftherest.ScalingyourclusterupisgenerallyastraightforwardoperationasMesoswillsimplylaunchanothertaskandbedonewithit.Scalingdownisabitmorecomplicatedasthedatafirsthastobemovedtosomeotherplacesothatwillnaturallytakesomewhatlonger.

Pleasenotethatscalingoperationsmightnotalwayswork.ForexampleiftheunderlyingMesosclusteriscompletelysaturatedwithitsrunningtasksscalingupwillnotbepossible.ScalingdownmightalsofailduetotheclusternotbeingabletomoveallshardsofaDBServertoanewdestinationbecauseofsizelimitations.BesuretochecktheoutputoftheArangoDBframework.

Deinstallation

DeinstallingArangoDBisabitmoredifficultasthereismuchstatebeingkeptintheMesosclusterwhichisnotautomaticallycleanedup.Todeinstallfromthecommandlineusethefollowingoneliner:

dcosarangodbuninstall;dcospackageuninstallarangodb

Thiswillfirstcleanupthestateintheclusterandthenuninstallarangodb.

arangodb-cleanup-framework

Shouldyouforgettocleanupthestateyoucandosolaterbyusingthearangodb-cleanup-frameworkcontainer.Otherwiseyoumightnotbeabletodeployanewarangodbinstallation.

ThecleanupframeworkwillannounceitselfasanormalArangoDB.MesoswillrecognizethisandofferallpersistentvolumesitstillhasforArangoDBtothisframework.Thecleanupframeworkwillthenproperlyfreethepersistentvolumes.Finallyitwillcleanupanystateleftinzookeeper(thecentralconfigurationmanagerinaMesoscluster).

Todeploythecleanupframework,followtheinstructionsinthegithubrepository.AfterdeploymentwatchtheoutputinthesandboxoftheMesoswebinterface.Afterawhilethereshouldn'tbeanypersistentresourceoffersanymoreaseverythingwascleanedup.AfterthatyoucandeletethecleanupframeworkagainviaMarathon.

ApacheMesosandMarathon

YoucanalsoinstallArangoDBonabareApacheMesosclusterprovidedthatMarathonisrunningonit.

Doingsohasthefollowingdownsides:

1. ManualMesosclustersetup2. Youneedtoimplementyourownservicediscovery3. Youaremissingthedcoscli4. Installationanddeinstallationaretedious5. YouneedtosetupsomekindofproxytunneltoaccessArangoDBfromtheoutside6. Sparsemonitoringcapabilities

HoweverthesearethingswhichdonotinfluenceArangoDBitselfandoperatingyourclusterlikethisisfullysupported.

InstallingviaMarathon

ToinstallArangoDBviamarathonyouneedaproperconfigfile:

{

Mesos,DC/OS

310

https://github.com/mesosphere/mesos-dns

https://github.com/arangodb/arangodb-cleanup-framework/

"id":"arangodb",

"cpus":0.25,

"mem":256.0,

"ports":[0,0,0],

"instances":1,

"args":[

"framework",

"--framework_name=arangodb",

"--master=zk://172.17.0.2:2181/mesos",

"--zk=zk://172.17.0.2:2181/arangodb",

"--user=",

"--principal=pri",

"--role=arangodb",

"--mode=cluster",

"--async_replication=true",

"--minimal_resources_agent=mem(*):512;cpus(*):0.25;disk(*):512",

"--minimal_resources_dbserver=mem(*):512;cpus(*):0.25;disk(*):1024",

"--minimal_resources_secondary=mem(*):512;cpus(*):0.25;disk(*):1024",

"--minimal_resources_coordinator=mem(*):512;cpus(*):0.25;disk(*):1024",

"--nr_agents=1",

"--nr_dbservers=2",

"--nr_coordinators=2",

"--failover_timeout=86400",

"--arangodb_image=arangodb/arangodb-mesos:3.1",

"--secondaries_with_dbservers=false",

"--coordinators_with_dbservers=false"

],

"container":{

"type":"DOCKER",

"docker":{

"image":"arangodb/arangodb-mesos-framework:3.1",

"network":"HOST"

}

},

"healthChecks":[

{

"protocol":"HTTP",

"path":"/framework/v1/health.json",

"gracePeriodSeconds":3,

"intervalSeconds":10,

"portIndex":0,

"timeoutSeconds":10,

"maxConsecutiveFailures":0

}

]

}

Carefullyreviewthesettings(especiallytheIPsandtheresources).ThenyoucandeploytoMarathon:

curl-XPOST-H"Content-Type:application/json"http://url-of-marathon/v2/[email protected]

AlternativelyusethewebinterfaceofMarathontodeployArangoDB.IthasaJSONmodeandyoucanusetheaboveconfigurationfile.

DeinstallationviaMarathon

AswithDC/OSyoufirstneedtoproperlycleanupanystateleftovers.

TheeasiestistosimplydeleteArangoDBandthendeploythecleanup-framework(seesectionarangodb-cleanup-framework).

Configurationoptions

TheArangodbMesosframeworkhasatonofdifferentoptionswhicharelistedanddescribedhere:https://github.com/arangodb/arangodb-mesos-framework/tree/3.1

Mesos,DC/OS

311

https://github.com/arangodb/arangodb-mesos-framework/tree/3.1

AutomaticnativeClusters

SimilarlytohowtheMesosframeworkarangesanArangoDBclusterinaDC/OSenvironmentforyou,arangodbcandothisforyouinaplainenvironment.

Byinvokingthefirstarangodbyoulaunchaprimarynode.Itwillbindanetworkport,andoutputthecommandsyouneedtocut'n'pasteintotheothernodes.Let'sreviewtheprocessofsuchastartuponthreehostsnamedh01,h02,andh03:

arangodb@h01~>arangodb--ownAddressh01:4000

2017/06/1214:59:38Startingarangodbversion0.5.0+git,build5f97368

2017/06/1214:59:38ServingasmasterwithID'52698769'onh01:4000...

2017/06/1214:59:38Waitingfor3serverstoshowup.

2017/06/1214:59:38Usethefollowingcommandstostartotherservers:

arangodb--dataDir=./db2--joinh01:4000

arangodb--dataDir=./db3--joinh01:4000

2017/06/1214:59:38Listeningon0.0.0.0:4000(h01:4000)

Soyoucutthelinesarangodb--data.dir=./db2--starter.join127.0.0.1andexecutethemfortheothernodes.Ifyourunitonanothernodeonyournetwork,replacethe--starter.join127.0.0.1bythepublicIPofthefirsthost.

arangodbh02~>arangodb--dataDir=./db2--joinh01:4000


2017/06/1214:48:50Contactingmasterh01:4000...

2017/06/1214:48:50Waitingfor3serverstoshowup...

2017/06/1214:48:50Listeningon0.0.0.0:4000(:4000)

arangodbh03~>arangodb--dataDir=./db3--joinh01:4000


2017/06/1214:48:50Contactingmasterh01:4000...

2017/06/1214:48:50Waitingfor3serverstoshowup...

2017/06/1214:48:50Listeningon0.0.0.0:4000(:4000)

Oncethetwootherprocessesjoinedthecluster,andstartedtheirArangoDBserverprocesses(thismaytakeawhiledependingonyoursystem),itwillinformyouwheretoconnecttheClusterfromaBrowser,shelloryourprogramm:

...

2017/06/1214:55:21coordinatorupandrunning.

Atthispointyoumayaccessyourclusterateithercoordinatorendpoint,http://h01:4002/,http://h02:4002/orhttp://h03:4002/.

AutomaticnativelocaltestClusters

Ifyouonlywantalocaltestcluster,youcanrunasinglestarterwiththe--starter.localargument.Itwillstarta3"machine"clusteronyourlocalPC.

arangodb--starter.local

Note.AlocalclusterisintendedonlyfortestpurposessinceafailureofasinglePCwillbringdowntheentirecluster.

AutomaticDockerClusters

ArangoDBStartercanalsobeusedtolaunchclustersbasedondockercontainers.Itsabitmorecomplicated,sinceyouneedtoprovideinformationaboutyourenvironmentthatcan'tbeautodetected.

IntheDockerworldyouneedtotakecareaboutwherepersistantdataisstored,sincecontainersareintendedtobevolatile.Weuseavolumenamedarangodb1here:

Generic&Docker

312

http://h01:4002/

http://h02:4002/

http://h03:4002/

https://github.com/arangodb-helper/arangodb#running-in-docker

dockervolumecreatearangodb1

(Youcanuseanytypeofdockervolumethatfitsyoursetupinstead.)

WethenneedtodeterminethetheIPofthedockerhostwhereyouintendtorunArangoDBstarteron.Dependingonyouroperatingsystemexecuteipaddr,ifconfigoripconfigtodetermineyourlocalipaddress.

192.168.1.0/24deveth0protokernelscopelinksrc192.168.1.32

SothisexampleusestheIP192.168.1.32:

dockerrun-it--name=adb1--rm-p8528:8528\

-varangodb1:/data\

-v/var/run/docker.sock:/var/run/docker.sock\

arangodb/arangodb-starter\

--starter.address=192.168.1.32

Itwillstartthemasterinstance,andcommandyoutostarttheslaveinstances:

Unabletofindimage'arangodb/arangodb-starter:latest'locally

latest:Pullingfromarangodb/arangodb-starter

Digest:sha256:b87d20c0b4757b7daa4cb7a9f55cb130c90a09ddfd0366a91970bcf31a7fd5a4

Status:Downloadednewerimageforarangodb/arangodb-starter:latest

2017/06/1213:26:14Startingarangodbversion0.7.1,buildf128884

2017/06/1213:26:14ServingasmasterwithID'46a2b40d'on192.168.1.32:8528...

2017/06/1213:26:14Waitingfor3serverstoshowup.

2017/06/1213:26:14Usethefollowingcommandstostartotherservers:

dockervolumecreatearangodb2&&\

dockerrun-it--name=adb2--rm-p8533:8528-varangodb2:/data\

-v/var/run/docker.sock:/var/run/docker.sockarangodb/arangodb-starter:0.7\

--starter.address=192.168.1.32--starter.join=192.168.1.32

dockervolumecreatearangodb3&&\

dockerrun-it--name=adb3--rm-p8538:8528-varangodb3:/data\

-v/var/run/docker.sock:/var/run/docker.sockarangodb/arangodb-starter:0.7\

--starter.address=192.168.1.32--starter.join=192.168.1.32

Onceyoustarttheotherinstances,itwillcontinuelikethis:

2017/05/1109:05:45Addedmaster'fc673b3b':192.168.1.32,portOffset:0

2017/05/1109:05:45Addednewpeer'e98ea757':192.168.1.32,portOffset:5

2017/05/1109:05:50Addednewpeer'eb01d0ef':192.168.1.32,portOffset:10

2017/05/1109:05:51Startingservice...

2017/05/1109:05:51Lookingforarunninginstanceofagentonport8531

2017/05/1109:05:51Startingagentonport8531

2017/05/1109:05:52Lookingforarunninginstanceofdbserveronport8530

2017/05/1109:05:52Startingdbserveronport8530

2017/05/1109:05:53Lookingforarunninginstanceofcoordinatoronport8529

2017/05/1109:05:53Startingcoordinatoronport8529

2017/05/1109:05:58agentupandrunning(version3.2.devel).

2017/05/1109:06:15dbserverupandrunning(version3.2.devel).

2017/05/1109:06:31coordinatorupandrunning(version3.2.devel).

Andatleastittellsyouwhereyoucanworkwithyourcluster:

2017/05/1109:06:31Yourclustercannowbeaccessedwithabrowserat`http://192.168.1.32:8529`or

2017/05/1109:06:31using`arangosh--server.endpointtcp://192.168.1.32:8529`.

Underthehood

Thefirstarangodbyouran(asshownabove)willbecomethemasterinyoursetup,the--starter.joinwillbetheslaves.

Generic&Docker

313

ThemasterdetermineswhichArangoDBserverprocessestolaunchonwhichslave,andhowtheyshouldcommunicate.Itwillthenlaunchtheserverprocessesandmonitorthem.Onceithasdetectedthatthesetupiscompleteyouwillgettheprompt.Themasterwillsavethesetupforsubsequentstarts.

MorecomplicatedsetupoptionscanbefoundinArangoDBStartersReadme.

Generic&Docker

314

https://github.com/arangodb-helper/arangodb#starting-an-arangodb-cluster-the-easy-way

AdvancedTopics

Incontrasttotheothertopicsinthischapterthatstrivetogetyousimplysetupinpreparedenvironments,Thefollowingchaptersdescribewhatsgoingonunderthehoodindetails,thecomponentsofArangoDBClusters,andhowthey'reputtogether:

StandaloneAgencyTestsetuponalocalmachineStartingprocessesondifferentmachinesLaunchinganArangoDBclusterusingDockercontainers

AdvancedTopics

315

LaunchingArangoDB'sstandalone"agency"

MultipleArangoDBinstancescanbedeployedasafault-tolerantdistributedstatemachine.

Whatisafault-tolerantstatemachineinthefirstplace?

Inmanyservicedeploymentsconsistingofarbitrarycomponentsdistributedovermultiplemachinesoneisfacedwiththechallengeofcreatingadependablecentralisedknowledgebaseorconfiguration.Implementationofsuchaserviceturnsouttobeoneofthemostfundamentalproblemsininformationengineering.Whileitmayseemasiftherealisationofsuchaserviceiseasilyconceivable,dependablityformulatesaparadoxononcomputernetworksperse.Ontheonehand,oneneedsadistributedsystemtoavoidasinglepointoffailure.Ontheotherhand,onehastoestablishconsensusamongthecomputersinvolved.

Consensusisthekeywordhereanditsrealisationonanetworkprovestobefarfromtrivial.Manypapersandconferenceproceedingshavediscussedandevaluatedthiskeychallenge.Twoalgorithms,historicallyfarapart,havebecomewidelypopular,namelyPaxosanditsderivativesandRaft.Discussingthemandtheirdifferences,althoughhighlyenjoyable,mustremainfarbeyondthescopeofthisdocument.Findthereferencestothemainpublicationsatthebottomofthispage.

AtArangoDB,wedecidedtoimplementRaftasitisarguablytheeasiertounderstandandthusimplement.Insimpleterms,Raftguaranteesthatalinearstreamoftransactions,isreplicatedinrealtimeamongagroupofmachinesthroughanelectedleader,whointurnmusthaveaccesstoandprojectleadershipuponanoverallmajorityofparticipatinginstances.InArangoDBweliketocalltheentiretyofthecomponentsofthereplicatedtransactionlog,thatisthemachinesandtheArangoDBinstances,whichconstitutethereplicatedlog,theagency.

Startup

Theagencymustconsistsofanoddnumberofagentsinordertobeabletoestablishanoverallmajorityandsomemeansfortheagentstobeabletofindoneanotheratstartup.

Themostobviouswaywouldbetoinformallagentsoftheaddressesandportsoftherest.Thishowever,ismoreinformationthanneeded.Forexample,itwouldsuffice,ifallagentswouldknowtheaddressandportofthenextagentinacyclicfashion.Anotherstraitforwardsolutionwouldbetoinformallagentsoftheaddressandportofsaythefirstagent.

Clearlyallcases,whichwouldformdisjunctsubsetsofagentswouldbreakorintheleastimpairthefunctionalityoftheagency.Fromthereontheagentswillgossipthemissinginformationabouttheirpeers.

Typically,oneachievesfairlyhighfault-tolerancewithlow,oddnumberofagentswhilekeepingthenecessarynetworktrafficataminimum.Itseemsthatthetypicalagencysizewillbeinrangeof3to7agents.

Thebelowcommandsstartupa3-hostagencyononephysical/logicalboxwithports8529,8530and8531fordemonstrationpurposes.Theadressofthefirstinstance,port8529,isknowntotheothertwo.Afteratmost2roundsofgossipping,thelast2agentswillhaveacompletepictureoftheirsurroundingandpersistitforthenextrestart.

./build/bin/arangod--agency.activatetrue--agency.size3--agency.my-addresstcp://localhost:8529--server.authenticationfal

se--server.endpointtcp://0.0.0.0:8529agency-8529

./build/bin/arangod--agency.activatetrue--agency.size3--agency.endpointtcp://localhost:8529--agency.my-addresstcp://loc

alhost:8530--server.authenticationfalse--server.endpointtcp://0.0.0.0:8530agency-8530

./build/bin/arangod--agency.activatetrue--agency.size3--agency.endpointtcp://localhost:8529--agency.my-addresstcp://loc

alhost:8531--server.authenticationfalse--server.endpointtcp://0.0.0.0:8531agency-8531

Theparameteragency.endpointisthekeyingredientforthesecondandthirdinstancestofindthefirstinstanceandthusformacompleteagency.Pleaserefertothetheshell-scriptscripts/startStandaloneAgency.shongithuborinthesourcedirectory.

Key-value-storeAPI

Theagencyshouldbeupandrunningwithinacoupleofseconds,duringwhichtheinstanceshavegossipedtheirwayintoknowingtheotheragentsandelectedaleader.ThepublicAPIcanbecheckedforthestateoftheconfiguration:

curl-slocalhost:8529/_api/agency/config

AdvancedTopics

316

{

"term":1,

"leaderId":"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98",

"lastCommitted":1,

"lastAcked":{

"ac129027-b440-4c4f-84e9-75c042942171":0.21,

"c54dbb8a-723d-4c82-98de-8c841a14a112":0.21,

"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98":0

},

"configuration":{

"pool":{

"ac129027-b440-4c4f-84e9-75c042942171":"tcp://localhost:8531",

"c54dbb8a-723d-4c82-98de-8c841a14a112":"tcp://localhost:8530",

"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98":"tcp://localhost:8529"

},

"active":[

"ac129027-b440-4c4f-84e9-75c042942171",

"c54dbb8a-723d-4c82-98de-8c841a14a112",

"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98"

],

"id":"f5d11cde-8468-4fd2-8747-b4ef5c7dfa98",

"agencysize":3,

"poolsize":3,

"endpoint":"tcp://localhost:8529",

"minping":0.5,

"maxping":2.5,

"supervision":false,

"supervisionfrequency":5,

"compactionstepsize":1000,

"supervisiongraceperiod":120

}

}

Tohighlightsomedetailsintheaboveoutputlookfor"term"and"leaderId".BotharekeyinformationaboutthecurrentstateoftheRaftalgorithm.Youmayhavenotedthatthefirstelectiontermhasestablishedarandomleaderfortheagency,whoisinchargeofreplicationofthestatemachineandforallexternalreadandwriterequestsuntilsuchtimethattheprocessgetsisolatedfromtheothertwosubsequenctlylosingitsleadership.

ReadandWriteAPIs

Generally,allreadandwriteaccessesaretransactionsmoreoveranyreadandwriteaccessmayconsistofmultiplesuchtransactionsformulatedasarraysofarraysinJSONdocuments.

Readtransaction

Anagencystartedfromscratchwilldealwiththesimplestqueryasfollows:

curl-Llocalhost:8529/_api/agency/read-d'[["/"]]'

[{}]

Theaboverequestforanemptykeyvaluestorewillreturnwithanemptydocument.Theinnerarraybracketswillaggregatearesultfrommultiplesourcesinthekey-value-storewhiletheouterarraywilldelivermultiplesuchaggregatedresults.Alsonotethe-Lcurlflag,whichallowstherequesttofollowredirectstothecurrentleader.

Considerthefollowingkey-value-store:

{

"baz":12,

"corge":{

"e":2.718281828459045,

"pi":3.14159265359

},

"foo":{

"bar":"HelloWorld"

},

AdvancedTopics

317

"qux":{

"quux":"HelloWorld"

}

}

Thefollowingarrayofreadtransactionswillyield:

curl-Llocalhost:8529/_api/agency/read-d'[["/foo","/foo/bar","/baz"],["/qux"]]'

[

{

"baz":12,

"foo":{

"bar":"HelloWorld"

}

},

{

"qux":{

"quux":"HelloWorld"

}

}

]

Notethattheresultisanarrayoftworesultsforthefirstandsecondreadtransactionsfromaboveaccordingly.Alsonotethattheresultsfromthefirstreadtransactionareaggregatedinto

{

"baz":12,

"foo":{

"bar":"HelloWorld"

}

}

Theaggregationisperformedon2levels:

1. /foo/bariseliminatedasasubsetof/foo2. Theresultsfrom/fooand/bararejoined

Thewordtransactionmeansherethatitisguaranteedthatallaggregationshappeninquasi-realtimeandthatnowriteaccesscouldhavehappenedinbetween.

Btw,thesametransactiononthevirginkey-valuestorewouldproduce[{},{}]

WriteAPI:

ThewriteAPImustunfortunatelybealittlemorecomplex.MultipleroadsleadtoRome:

curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"bar"}}]]'

curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"baz"}}]]'

curl-Llocalhost:8529/_api/agency/write-d'[[{"/foo":{"op":"push","new":"qux"}}]]'

and

curl-Llocalhost:8529/_api/agency/write-d'[[{"foo":["bar","baz","qux"]}]]'

areequivalentforexampleandwillcreateandfillanarrayat/foo.Here,again,theoutermostarrayisthecontainerforthetransactionarrays.

WedocumententacompleteguideoftheAPIintheAPIsection.

AdvancedTopics

318

LaunchinganArangoDBclusterfortesting

AnArangoDBclusterconsistsofseveralrunningtasks(orserverprocesses)whichformthecluster.ArangoDBitselfwon'tstartormonitoranyofthesetasks.Soitwillneedsomekindofsupervisorwhichismonitoringandstartingthesetasks.ForproductionusagewerecommendusingApacheMesosastheclustersupervisor.

HoweverstartingaclustermanuallyispossibleandisaveryeasymethodtogetafirstimpressionofwhatanArangoDBclusterlookslike.

Theeasiestwaytostartalocalclusterfortestingpurposesistorunscripts/startLocalCluster.shfromacloneofthesourcerepositoryaftercompilingArangoDBfromsource(seeinstructionsinthefileREADME_maintainers.mdintherepository.Thiswillstart1Agency,2DBServersand1Coordinator.Tostoptheclusterissuescripts/stopLocalCluster.sh.

ThissectionwilldiscusstherequiredparametersforeveryroleinanArangoDBcluster.BesuretoreadtheArchitecturedocumentationtogetabasicunderstandingoftheunderlyingarchitectureandtheinvolvedrolesinanArangoDBcluster.

Inthefollowingsectionswewillgothroughtherelevantoptionsperrole.

Agency

Tostartupanagencyyoufirsthavetoactivateit.Thisisdonebyproviding--agency.activatetrue.

Tostartuptheagencyinitsfaulttolerantmodesetthe--agency.sizeto3.Youwillthenhavetoprovideatleast3agentsbeforetheagencywillstartoperation.

Duringinitializationtheagentshavetofindeachother.Todosoprovideatleastonecommon--agency.endpoint.Theagentswillthencoordinatestartupthemselves.Theywillannouncethemselveswiththeirexternaladdresswhichmaybespecifiedusing--agency.my-address.ThisisrequiredinbridgeddockersetupsorNATedenvironments.

Soinsummarythisiswhatyourstartupmightlooklike:

AdvancedTopics

319

https://github.com/ArangoDB/ArangoDB

arangod--server.endpointtcp://0.0.0.0:5001--agency.my-address=tcp://127.0.0.1:5001--server.authenticationfalse--agency.ac

tivatetrue--agency.size3--agency.endpointtcp://127.0.0.1:5001--agency.supervisiontrue--database.directoryagency1&





Ifyouarehappywithasingleagent,thensimplyuseasinglecommandlikethis:

arangod--server.endpointtcp://0.0.0.0:5001--server.authenticationfalse--agency.activatetrue--agency.size1--agency.endp

ointtcp://127.0.0.1:5001--agency.supervisiontrue--database-directoryagency1&

Furthermore,inthefollowingsectionswhen--cluster.agency-addressisusedmultipletimestospecifyallthreeagentaddresses,justuseasingleoption--cluster.agency.addresstcp://127.0.0.1:5001instead.

CoordinatorsandDBServers

Thesetworolesshareacommonsetofrelevantoptions.Firstyoushouldspecifytheroleusing--cluster.my-role.ThiscaneitherbePRIMARY(adatabaseserver)orCOORDINATOR.Furthermoreprovidetheexternalendpoint(IPandport)ofthetaskvia--cluster.my-address.

Thefollowingisafull-exampleofwhatitmightlooklike:

arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8529--cluster.my-addresstcp://127.0.0.1:8529--cluster.

my-rolePRIMARY--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.agency-

endpointtcp://127.0.0.1:5003--database.directoryprimary1&


my-rolePRIMARY--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.agency-

endpointtcp://127.0.0.1:5003--database.directoryprimary2&


my-roleCOORDINATOR--cluster.agency-endpointtcp://127.0.0.1:5001--cluster.agency-endpointtcp://127.0.0.1:5002--cluster.age

ncy-endpointtcp://127.0.0.1:5003--database.directorycoordinator&

Noteinparticularthattheendpointdescriptionsgivenunder--cluster.my-addressand--cluster.agency-endpointmustnotusetheIPaddress0.0.0.0becausetheymustcontainanactualaddressthatcanberoutedtothecorrespondingserver.The0.0.0.0in--server.endpointsimplymeansthattheserverbindsitselftoallavailablenetworkdeviceswithallavailableIPaddresses.

UponregisteringwiththeagencyduringstartuptheclusterwillassignanIDtoeveryserver.ThegeneratedIDwillbeprintedouttothelogorcanbeaccessedviathehttpAPIbycallinghttp://server-address/_admin/server/id.

YouhavenowlaunchedacompleteArangoDBclusterandcancontactitscoordinatorattheendpointtcp://127.0.0.1:8531,whichmeansthatyoucanreachthewebUIunderhttp://127.0.0.1:8531.

AdvancedTopics

320

LaunchinganArangoDBclusteronmultiplemachines

Essentially,onecanusethemethodfromtheprevioussectiontostartanArangoDBclusteronmultiplemachinesaswell.Theonlychangesarethatonehastoreplacealllocaladdresses127.0.0.1bytheactualIPaddressofthecorrespondingserver.

IfweassumethatyouwanttostartyouArangoDBclusteronthreedifferentmachineswithIPaddresses

192.168.1.1

192.168.1.2

192.168.1.3

thenthecommandsyouhavetouseare(youcanusehostnamesiftheycanberesolvedtoIPaddressesonallmachines):

On192.168.1.1:

sudoarangod--server.endpointtcp://0.0.0.0:5001--agency.my-addresstcp://192.168.1.1:5001--server.authenticationfalse--ag

ency.activatetrue--agency.size3--agency.supervisiontrue--database.directoryagency

On192.168.1.2:


ency.activatetrue--agency.size3--agency.supervisiontrue--database.directoryagency

On192.168.1.3:


ency.activatetrue--agency.size3--agency.endpointtcp://192.168.1.1:5001--agency.endpointtcp://192.168.1.2:5001--agency.e

ndpointtcp://192.168.1.3:5001--agency.supervisiontrue--database.directoryagency

On192.168.1.1:

sudoarangod--server.authentication=false--server.endpointtcp://0.0.0.0:8529--cluster.my-addresstcp://192.168.1.1:8529--c

luster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clus

ter.agency-endpointtcp://192.168.1.3:5001--database.directoryprimary1&

On192.168.1.2:

sudoarangod--server.authentication=false--server.endpointtcp://0.0.0.0:8530--cluster.my-addresstcp://192.168.1.2:8530--c

luster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clus

ter.agency-endpointtcp://192.168.1.3:5001--database.directoryprimary2&

On192.168.1.3:

arangod--server.authentication=false--server.endpointtcp://0.0.0.0:8531--cluster.my-addresstcp://192.168.1.3:8531--cluste

r.my-roleCOORDINATOR--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agency-endpointtcp://192.168.1.2:5001--clust

er.agency-endpointtcp://192.168.1.3:5001--database.directorycoordinator&

Obviously,itwouldnolongerbenecessarytousedifferentportnumbersondifferentservers.Wehavechosentokeepallportnumbersincomparisontothelocalsetuptominimizethenecessarychanges.

Afterhavingswallowedtheselongishcommands,wehopethatyouappreciatethesimplicityofthesetupwithApacheMesosandDC/OS.

AdvancedTopics

321

ArangoDBClusterandDocker

Networking

AbitofextracarehastobeinvestedduetothewayinwhichDockerisolatesitsnetwork.Bydefaultitfullyisolatesthenetworkandbydoingsoanendpointlike--server.endpointtcp://0.0.0.0:8529willonlybindtoallinterfacesinsidetheDockercontainerwhichdoesnotincludeanyexternalinterfaceonthehostmachine.ThismaybesufficientifyoujustwanttoaccessitlocallybutincaseyouwanttoexposeittotheoutsideyoumustfacilitateDockersportforwardingusingthe-pcommandlineoption.BesuretochecktheofficialDockerdocumentation.

Tosimplymakearangodbavailableonallhostinterfacesonport8529:

dockerrun-p8529:8529-eARANGO_NO_AUTH=1arangodb

AnotherpossibilityistostartDockervianetworkmodehost.Thisispossiblebutgenerallynotrecommended.TodoitanywaychecktheDockerdocumentationfordetails.

DockerandClustertasks

TostarttheclusterviaDockerisbasicallythesameasstartinglocallyoronmultiplemachines.Howeverjustlikewiththesinglenetworkingimagewewillfacenetworkingissues.Youcansimplyusethe-pflagtomaketheindividualtaskavailableonthehostmachineoryoucoulduseDocker'slinkstoenabletaskintercommunication.

PleasenotethattherearesomeflagsthatspecifyhowArangoDBcanreachataskfromtheoutside.Theseareveryimportantandbuiltforthisexactusecase.Anexampleconfigurationmightlooklikethis:

dockerrun-eARANGO_NO_AUTH=1-p192.168.1.1:10000:8529arangodb/arangodbarangod--server.endpointtcp://0.0.0.0:8529--clust

er.my-addresstcp://192.168.1.1:10000--cluster.my-rolePRIMARY--cluster.agency-endpointtcp://192.168.1.1:5001--cluster.agen

cy-endpointtcp://192.168.1.2:5002--cluster.agency-endpointtcp://192.168.1.3:5003

ThiswillstartaprimaryDBserverwithinaDockercontainerwithanisolatednetwork.WithintheDockercontaineritwillbindtoallinterfaces(thiswillbe127.0.0.1:8529andsomeinternalDockeriponport8529).Bysupplying-p192.168.1.1:10000:8529weareestablishingaportforwardingfromourlocalIP(192.168.1.1port10000inthisexample)toport8529insidethecontainer.Withinthecommandwearetellingarangodhowitcanbereachedfromtheoutside--cluster.my-addresstcp://192.168.1.1:10000.ThisinformationwillbeforwardedtotheagencysothattheothertasksinyourclustercanseehowthisparticularDBServermaybereached.

AdvancedTopics

322

https://docs.docker.com/engine/reference/run/

https://docs.docker.com/engine/reference/run/

Datacentertodatacenterreplication.

About

Atsomepointinthegrowsofadatabase,therecomesaneedforreplicatingitacrossmultipledatacenters.

Reasonsforthatcanbe:

Fallbackincaseofadisasterinonedatacenter.RegionalavailabilitySeparationofconcerns

Andmanymore.

ThistutorialdescribeswhattheArangoSyncdatacentertodatacenterreplicationsolution(ArangoSyncfromnowon)offers,whentouseit,whennottouseitandhowtoconfigure,operate,troubleshootit&keepitsafe.

Whatisit

ArangoSyncisasolutionthatenablesyoutoasynchronouslyreplicatetheentirestructureandcontentinanArangoDBclusterinoneplacetoaclusterinanotherplace.Typicallyitisusedfromonedatacentertoanother.Itisnotasolutionforreplicatingsingleserverinstances.

ThereplicationdonebyArangoSyncinasynchronous.Thatmeansthatwhenaclientiswritingdataintothesourcedatacenter,itwillconsidertherequestfinishedbeforethedatahasbeenreplicatedtotheotherdatacenter.Thetimeneededtocompletelyreplicatechangestotheotherdatacenteristypicallyintheorderofseconds,butthiscanvarysignificantlydependingonload,network&computercapacity.

ArangoSyncperformsreplicationinasingledirectiononly.ThatmeansthatyoucanreplicatedatafromclusterAtoclusterBorfromclusterBtoclusterA,butneveratthesametime.Datamodifiedinthedestinationclusterwillbelost!

Replicationisacompletelyautonomousprocess.Onceitisconfigureditisdesignedtorun24/7withoutfrequentmanualintervention.Thisdoesnotmeanthatitrequiresnomaintenanceorattentionatall.Aswithanydistributedsystemsomeattentionisneededtomonitoritsoperationandkeepitsecure(e.g.certificate&passwordrotation).

Onceconfigured,ArangoSyncwillreplicatebothstructureanddataofanentirecluster.Thismeansthatthereisnoneedtomakeadditionalconfigurationchangeswhenadding/removingdatabasesorcollections.Alsometadatasuchasusers,foxxapplication&jobsareautomaticallyreplicated.

Whentouseit...andwhennot

ArangoSyncisagoodsolutioninallcaseswhereyouwanttoreplicatedatafromoneclustertoanotherwithouttherequirementthatthedataisavailableimmediatelyintheothercluster.

ArangoSyncisnotagoodsolutionwhenoneofthefollowingapplies:

YouwanttoreplicatedatafromclusterAtoclusterBandfromclusterBtoclusterAatthesametime.Youneedsynchronousreplicationbetween2clusters.ThereisnonetworkconnectionbetweeclusterAandB.Youwantcompletecontroloverwhichdatabase,collection&documentsarereplicateandwhichnot.

Requirements

TouseArangoSyncyouneedthefollowing:

Twodatacenters,eachrunninganArangoDBEnterprisecluster,version3.3orhigher.

MultipleDatacenters

323

Anetworkconnectionbetweenbothdatacenterswithaccessibleendpointsforseveralcomponents(seeindividualcomponentsfordetails).TLScertificatesforArangoSyncmasterinstances(canbeself-signed).TLScertificatesforKafkabrokers(canbeself-signed).Optional(butrecommended)TLScertificatesforArangoDBclusters(canbeself-signed).ClientcertificatesCAforArangoSyncmasters(typicallyself-signed).ClientcertificatesforArangoSyncmasters(typicallyself-signed).Atleast2instancesoftheArangoSyncmasterineachdatacenter.OneinstancesoftheArangoSyncworkeroneverymachineineachdatacenter.

Note:Inseveralplacesyouwillneeda(x509)certificate.Thecertificatessectionbelowprovidesmoreguidanceforcreatingandrenewingthesecertificates.

Besidestheabovelist,youprobablywanttousethefollowing:

Anorchestratortokeepallcomponentsrunning.Inthistutorialwewillusesystemdasanexample.Alogfilecollectorforcentralizedcollection&accesstothelogsofallcomponents.Ametricscollector&viewingsolutionsuchasPrometheus+Grafana.

Deployment

Inthefollowingparagraphsyou'lllearnhowtodeployallthecomponentsneededfordatacentertodatacenterreplication.

ArangoDBcluster

ThereareseveralwaystostartanArangoDBcluster.InthistutorialwewillfocusonourrecommendedwaytostartArangoDB:theArangoDBstarter.

Datacentertodatacenterreplicationrequirestherocksdbstorageengine.Inthistutorialtheexamplesetupwillhaverocksdbenabled.Ifyouchoosetodeploywithadifferentstrategykeepinmindtosetthestorageengine.

AlsoseeotherpossibilitiestodeployanArangoDBcluster.

Thestartersimplifiesthingsfortheoperatorandwillcoordinateadistributedclusterstartupacrossseveralmachinesandassignclusterrolesautomatically.

Whenstartedonseveralmachinesandenoughmachineshavejoined,thestarterswillstartagents,coordinatorsanddbserversonthesemachines.

Whenrunningthestarterwillsuperviseitschildtasks(namelycoordinators,dbserversandagents)andrestartthemincaseoffailure.

Tostarttheclusterusingasystemdunitfileusethefollowing:

[Unit]

Description=RuntheArangoDBStarter

After=network.target

[Service]

Restart=on-failure

EnvironmentFile=/etc/arangodb.env

EnvironmentFile=/etc/arangodb.env.local

Environment=DATADIR=/var/lib/arangodb/cluster

ExecStartPre=/usr/bin/sh-c"mkdir-p${DATADIR}"

ExecStart=/usr/bin/arangodb\

--starter.address=${PRIVATEIP}\

--starter.data-dir=${DATADIR}\

--starter.join=${STARTERENDPOINTS}\

--server.storage-engine=rocksdb\

--auth.jwt-secret=${CLUSTERSECRETPATH}

TimeoutStopSec=60

[Install]

WantedBy=multi-user.target

Notethatwesetrocksdbintheunitservicefile.

MultipleDatacenters

324

Clusterauthentication

Thecommunicationbetweentheclusternodesuseatoken(JWT)toauthenticate.Thismustbesharedbetweenclusternodes.

Sharingsecretsisobviouslyaverydelicatetopic.Theaboveworkflowassumesthattheoperatorwillputasecretinafilenamed${CLUSTERSECRETPATH}.

WerecommendtouseadedicatedsystemformanagingsecretslikeHashiCorps'VaultorthesecretmanagementofDC/OS.

Requiredports

Assoonasenoughmachineshavejoined,thestarterwillbeginstartingagents,coordinatorsanddbservers.

Eachofthesetasksneedsaporttocommunicate.Pleasemakesurethatthefollowingportsareavailableonallmachines:

8529forcoordinators8530fordbservers8531foragents

Thestarteritselfwilluseport8528.

Kafka&Zookeeper

HowtodeployzookeeperHowtodeploykafkaAccessibleports

SyncMaster

TheSyncMasterisresponsibleformanagingallsynchronization,creatingtasksandassigningthosetoworkers.Atleast2instancesmutsbedeployedineachdatacenter.Oneinstancewillbethe"leader",theotherwillbeaninactiveslave.Whentheleaderisgoneforashortwhile,oneoftheotherinstanceswilltakeover.

Withclustersofasignificantsize,thesyncmasterwillrequireasignificantsetofresources.Thereforeitisrecommendedtodeploysyncmastersontheirownservers,equipedwithsufficientCPUpowerandmemorycapacity.

Tostartasyncmasterusingasystemdservice,useaunitlikethis:

[Unit]

Description=RunArangoSyncinmastermode


[Service]

Restart=on-failure



ExecStart=/usr/sbin/arangosyncrunmaster\

--log.level=debug\

--cluster.endpoint=${CLUSTERENDPOINTS}\

--cluster.jwtSecret=${CLUSTERSECRET}\

--server.keyfile=${CERTIFICATEDIR}/tls.keyfile\

--server.client-cafile=${CERTIFICATEDIR}/client-auth-ca.crt\

--server.endpoint=https://${PUBLICIP}:${MASTERPORT}\

--server.port=${MASTERPORT}\

--master.jwtSecret=${MASTERSECRET}\

--mq.type=kafka\

--mq.kafka-addr=${KAFKAENDPOINTS}\

--mq.kafka-client-keyfile=${CERTIFICATEDIR}/kafka-client.key\

--mq.kafka-cacert=${CERTIFICATEDIR}/tls-ca.crt\

TimeoutStopSec=60

[Install]


ThesyncmasterneedsaTLSservercertificateandaIfyouwanttheservicetocreateaTLScertificate&clientauthenticationcertificate,forauthenticatingwithsyncmastersinanotherdatacenter,foreverystart,addthistotheServicesection.

MultipleDatacenters

325

ExecStartPre=/usr/bin/sh-c"mkdir-p${CERTIFICATEDIR}"

ExecStartPre=/usr/sbin/arangosynccreatetlskeyfile\

--cacert=${CERTIFICATEDIR}/tls-ca.crt\

--cakey=${CERTIFICATEDIR}/tls-ca.key\

--keyfile=${CERTIFICATEDIR}/tls.keyfile\

--host=${PUBLICIP}\

--host=${PRIVATEIP}\

--host=${HOST}

ExecStartPre=/usr/sbin/arangosynccreateclient-authkeyfile\

--cacert=${CERTIFICATEDIR}/tls-ca.crt\

--cakey=${CERTIFICATEDIR}/tls-ca.key\

--keyfile=${CERTIFICATEDIR}/kafka-client.key\

--host=${PUBLICIP}\

--host=${PRIVATEIP}\

--host=${HOST}

ThesyncmastermustbereachableonaTCPport${MASTERPORT}(usedwith--server.portoption).Thisportmustbereachablefrominsidethedatacenter(bysyncworkersandoperations)andfrominsideoftheotherdatacenter(bysyncmastersintheotherdatacenter).

SyncWorkers

TheSyncWorkerisresponsibleforexecutingsynchronizationtasks.Foroptimalperformanceatleast1workerinstancemustbeplacedoneverymachinethathasanArangoDBdbserverrunning.Thisensuresthattaskscanbeexecutedwithminimalnetworktrafficoutsideofthemachine.

SincesyncworkerswillautomaticallystoponcetheirTLSservercertificateexpires(whichissetto2yearsbydefault),itisrecommendedtorunatleast2instancesofaworkeroneverymachineinthedatacenter.Thatway,taskscanstillbeassignedinthemostoptimalway,evenwhenaworkerintemporarilydownforarestart.

Tostartasyncworkerusingasystemdservice,useaunitlikethis:

[Unit]

Description=RunArangoSyncinworkermode


[Service]

Restart=on-failure



Environment=PORT=8729

ExecStart=/usr/sbin/arangosyncrunworker\

--log.level=debug\

--server.port=${PORT}\

--server.endpoint=https://${PRIVATEIP}:${PORT}\

--master.endpoint=${MASTERENDPOINTS}\

--master.jwtSecret=${MASTERSECRET}

TimeoutStopSec=60

[Install]


ThesyncworkermustbereachableonaTCPport${PORT}(usedwith--server.portoption).Thisportmustbereachablefrominsidethedatacenter(bysyncmasters).

Prometheus&Grafana(optional)

ArangoSyncprovidesmetricsinaformatsupportedbyPrometheus.WealsoprovideastandardsetofdashboardsforviewingthosemetricsinGrafana.

Ifyouwanttousethesetools,gototheirwebsitesforinstructionsonhowtodeploythem.

Afterdeployment,youmustconfigureprometheususingaconfigurationfilethatinstructsitaboutwhichtargetstoscrape.ForArangoSyncyoushouldconfigurescrapetargetsforallsyncmastersandallsyncworkers.Todoso,youcanuseaconfigurationsuchasthis:

global:

scrape_interval:10s#scrapetargetsevery10seconds.

MultipleDatacenters

326

https://prometheus.io

https://grafana.org

scrape_configs:

#Scrapsyncmasters

-job_name:'sync_master'

scheme:'https'

bearer_token:"${MONITORINGTOKEN}"

tls_config:

insecure_skip_verify:true

static_configs:

-targets:

-"${IPMASTERA1}:8629"

-"${IPMASTERA2}:8629"

-"${IPMASTERB1}:8629"

-"${IPMASTERB2}:8629"

labels:

type:"master"

relabel_configs:

-source_labels:[__address__]

regex:${IPMASTERA1}\:8629|${IPMASTERA2}\:8629

target_label:dc

replacement:A


regex:${IPMASTERB1}\:8629|${IPMASTERB2}\:8629

target_label:dc

replacement:B


regex:${IPMASTERA1}\:8629|${IPMASTERB1}\:8629

target_label:instance

replacement:1


regex:${IPMASTERA2}\:8629|${IPMASTERB2}\:8629


replacement:2

#Scrapsyncworkers

-job_name:'sync_worker'

scheme:'https'

bearer_token:"${MONITORINGTOKEN}"

tls_config:

insecure_skip_verify:true

static_configs:

-targets:

-"${IPWORKERA1}:8729"

-"${IPWORKERA2}:8729"

-"${IPWORKERB1}:8729"

-"${IPWORKERB2}:8729"

labels:

type:"worker"

relabel_configs:


regex:${IPWORKERA1}\:8729|${IPWORKERA2}\:8729

target_label:dc

replacement:A


regex:${IPWORKERB1}\:8729|${IPWORKERB2}\:8729

target_label:dc

replacement:B


regex:${IPWORKERA1}\:8729|${IPWORKERB1}\:8729


replacement:1


regex:${IPWORKERA2}\:8729|${IPWORKERB2}\:8729


replacement:2

Note:Theaboveexampleassumes2datacenters,with2syncmasters&2syncworkersperdatacenter.Youhavetoreplaceall${...}variablesintheaboveconfigurationwithapplicablevaluesfromyourenvironment.

Configuration

MultipleDatacenters

327

OnceallcomponentsoftheArangoSyncsolutionhavebeendeployedandarerunningproperly,ArangoSyncwillnotautomaticallyreplicatedatabasestructureandcontent.Forthat,itisisneededtoconfiguresynchronization.

Toconfiguresynchronization,youneedthefollowing:

Theendpointofthesyncmasterinthetargetdatacenter.Theendpointofthesyncmasterinthesourcedatacenter.Acertificate(inkeyfileformat)usedforclientauthenticationofthesyncmaster(withthesyncmasterinthesourcedatacenter).ACAcertificate(publickeyonly)forverifyingtheintegrityofthesyncmasters.Ausername+passwordpair(orclientcertificate)forauthenticatingtheconfigurerequirewiththesyncmaster(inthetargetdatacenter)

Withthatinformation,run:

arangosyncconfiguresync\

--master.endpoint=<endpointsofsyncmastersintargetdatacenter>\

--master.keyfile=<keyfileofofsyncmastersintargetdatacenter>\

--source.endpoint=<endpointsofsyncmastersinsourcedatacenter>\

--source.cacert=<publickeyofCAcertificateusedtoverifysyncmasterinsourcedatacenter>\

--auth.user=<usernameusedforauthenticationofthiscommand>\

--auth.password=<passwordofauth.user>

Thecommandwillfinishquickly.Afterwardsitwilltakesometimeuntiltheclustersinbothdatacentersareinsync.

Usethefollowingcommandtoinspectthestatusofthesynchronizationofadatacenter:

arangosyncgetstatus\

--master.endpoint=<endpointsofsyncmastersindatacenterofinterest>\


--auth.password=<passwordofauth.user>\

-v

Note:Invokingthiscommandonthetargetdatacenterwillreturndifferentresultsfrominvokingitonthesourcedatacenter.Youneedinsightinbothresultstogeta"completepicture".

Wherethegetstatuscommandgivesinsightinthestatusofsynchronization,therearemoredetailedcommandstogiveinsightintasks&registeredworkers.

Usethefollowingcommandtogetalistofallsynchronizationtasksinadatacenter:

arangosyncgettasks\




-v

Usethefollowingcommandtogetalistofallmastersinadatacenterandknowwhichmasteristhecurrentleader:

arangosyncgetmasters\




-v

Usethefollowingcommandtogetalistofallworkersinadatacenter:

arangosyncgetworkers\




-v

Stopsynchronization

MultipleDatacenters

328

Ifyounolongerwanttosynchronizedatafromasourcetoatargetdatacenteryoumuststopit.Todoso,runthefollowingcommand:

arangosyncstopsync\




Thecommandwillwaituntilsynchronizationhascompletelystoppedbeforereturning.Ifthesynchronizationisnotcompletelystoppedwithinareasonableperiod(2minutesbydefault)thecommandwillfail.

Ifthesourcedatacenterisnolongeravailableitisnotpossibletostopsynchronizationinagracefulmanner.Ifthathappensabortthesynchronizationwiththefollowingcommand:

arangosyncabortsync\




Ifthesourcedatacenterrecoversafteranabortsynchasbeenexecuted,itisneededto"cleanup"ArangoSyncinthesourcedatacenter.Todoso,executethefollowingcommand:

arangosyncabortoutgoingsync\

--master.endpoint=<endpointsofsyncmastersinsourcedatacenter>\



Reversingsynchronizationdirection

Ifyouwanttoreversethedirectionofsynchronization(e.g.afterafailureindatacenterAandyouswitchedtothedatacenterBforfallback),youmustfirststop(orabort)theoriginalsynchronization.

Oncethatisfinished(andcleanuphasbeenappliedincaseofabort),youmustnowconfigurethesynchronizationagain,butwithswappedsource&targetsettings.

Operations&Maintenance

ArangoSyncisadistributedsystemwithalotdifferentcomponents.Aswithanysuchsystem,itrequiressome,butnotalot,ofoperationalsupport.

Whatmeansareavailabletomonitorstatus

AllofthecomponentsofArangoSyncprovidemeanstomonitortheirstatus.Belowyou'llfindanoverviewpercomponent.

Syncmaster&workers:Thearangosyncserversrunningaseithermasterorworker,provide:

AstatusAPI,seearangosyncgetstatus.Makesurethatallstatusesreportrunning.Forevenmoredetailthefollowingcommandsarealsoavailable:arangosyncgettasks,arangosyncgetmasters&arangosyncgetworkers.Alogonthestandardoutput.Loglevelscanbeconfiguredusing--log.levelsettings.AmetricsAPIGET/metrics.ThisAPIiscompatiblewithPrometheus.SampleGrafanadashboardsforinspectingthesemetricsareavailable.

ArangoDBcluster:ThearangodserversthatmakeuptheArangoDBclusterprovide:

Alogfile.Thisisconfigurablewithsettingswithalog.prefix.E.g.--log.output=file://myLogFileor--log.level=info.AstatisticsAPIGET/_admin/statistics

Kafkacluster:Thekafkabrokersprovide:

Alogfile,seesettingswithlog.prefixinitsserver.propertiesconfigurationfile.Zookeeper:Thezookeeperagentsprovide:

Alogonstandardoutput.

MultipleDatacenters

329

Whattolookforwhilemonitoringstatus

TheveryfirstthingtodowhenmonitoringthestatusofArangoSyncistolookintothestatusprovidedbyarangosyncgetstatus...-v.Whennoteverythingisintherunningstate(onbothdatacenters),thisisanindicationthatsomethingmaybewrong.Incasethathappens,giveitsometime(incrementalsynchronizationmaytakequitesometimeforlargecollections)andlookatthestatusagain.Ifthestatusesdonotchange(orchange,butnotreachrunning)itistimetoinspectsthemetrics&logfiles.Whenthemetricsorlogsseemtoindicateaprobleminasyncmasterorworker,itissafetorestartit,aslongasonly1instanceisrestartedatatime.Giverestartedinstancessometimeto"catchup".

Whattodowhenproblemsremain

Whenaproblemremainsandrestartingmasters/workersdoesnotsolvetheproblem,contactsupport.Makesuretoincludeprovidesupportwiththefollowinginformation:

Outputofarangosyncgetversion...onbothdatacenters.Outputofarangosyncgetstatus...-vonbothdatacenters.Outputofarangosyncgettasks...-vonbothdatacenters.Outputofarangosyncgetmasters...-vonbothdatacenters.Outputofarangosyncgetworkers...-vonbothdatacenters.LogfilesofallcomponentsAcompletedescriptionoftheproblemyouobservedandwhatyoudidtoresolveit.

HowtomonitorstatusofArangoSync

HowtokeepitaliveWhattodoincaseoffailuresorbugs

Whattodowhenasourcedatacenterisdown

WhenyouuseArangoSyncforbackupofyourclusterfromonedatacentertoanotherandthesourcedatacenterhasacompleteoutage,youmayconsiderswitchingyourapplicationstothetarget(backup)datacenter.

Thisiswhatyoumustdointhatcase.

1. Stopconfigurationusingarangosyncstopsync....Whenthesourcedatacenteriscompletelyunresponsivethiswillnotsucceed.Inthatcaseusearangosyncabortsync....SeeConfigurationforhowtocleanupthesourcedatacenterwhenitbecomesavailableagain.

2. Verifythatconfigurationhascompletelystoppedusingarangosyncgetstatus...-v.3. Reconfigureyourapplicationstousethetarget(backup)datacenter.

Whentheoriginalsourcedatacenterisrestored,youmayswitchrolesandmakeitthetargetdatacenter.Todoso,usearangosyncconfiguresync...asdescribedinConfiguration.

Whattodoincaseofaplannednetworkoutage.

AllArangoSynctaskssendoutheartbeatmessagesouttotheotherdatacentertoindicate"itisstillalive".Theotherdatacenterassumestheconnectionis"outofsync"whenitdoesnotreceiveanymessagesforacertainperiodoftime.

Ifyou'replanningsomesortofmaintenancewhereyouknowtheconnectivitywillbelostforsometime(e.g.3hours),youcanprepareArangoSyncforthatsuchthatitwillholdofre-synchronizationforagivenperiodoftime.

Todoso,onbothdatacenters,run:

arangosyncsetmessagetimeout\

--master.endpoint=<endpointsofsyncmastersinthedatacenter>\



3h

ThelastargumentistheperiodthatArangoSyncshouldhold-ofresynchronizationfor.Thiscanbeminutes(e.g.15m)orhours(e.g.3h).

MultipleDatacenters

330

Ifmaintenanceistakinglongerthanexpected,youcanusethesamecommandtheextendtheholdofperiod(e.g.to4h).

Afterthemaintenance,usethesamecommandrestoretheholdofperiodtoitsdefaultof1h.

Whattodoincaseofadocumentthatexceedsthemessagequeuelimits.

Ifyouinsert/updateadocumentinacollectionandthesizeofthatdocumentislargerthanthemaximummessagesizeofyourmessagequeue,thecollectionwillnolongerbeabletosynchronize.Itwillgointoafailedstate.

Torecoverfromthat,firstremovethedocumentfromtheArangoDBclusterinthesourcedatacenter.Afterthat,foreachfailedshard,run:

arangosyncresetfailedshard\

--master.endpoint=<endpointsofsyncmastersinthedatacenter>\



--database=<nameofthedatabase>\

--collection=<nameofthecollection>\

--shard=<indexoftheshard(startingat0)>

Afterthiscommand,anewsetoftaskswillbestartedtosynchronizetheshard.Itcantakesometimefortheshardtoreachrunningstate.

Metrics

ArangoSync(master&worker)providemetricsthatcanbeusedformonitoringtheArangoSyncsolution.ThesemetricsareavailableusingthefollowingHTTPSendpoints:

GET/metrics:ProvidesmetricsinaformatsupportedbyPrometheus.GET/metrics.json:ProvidesthesamemetricsinJSONformat.

Bothendpointsincludehelpinformationpermetrics.

Note:Bothendpointsrequireauthentication.Besidestheusualauthenticationmethodstheseendpointsarealsoaccessibleusingaspecialbearertokenspecifiedusingthe--monitoring.tokencommandlineoption.

ThePrometheusoutput(/metrics)lookslikethis:

...

#HELParangosync_master_worker_registrationsTotalnumberofregistrations

#TYPEarangosync_master_worker_registrationscounter

arangosync_master_worker_registrations2

#HELParangosync_master_worker_storageNumberoftimesworkerinfoisstored,loaded

#TYPEarangosync_master_worker_storagecounter

arangosync_master_worker_storage{kind="",op="save",result="success"}20

arangosync_master_worker_storage{kind="empty",op="load",result="success"}1

...

TheJSONoutput(/metrics.json)lookslikethis:

{

...

"arangosync_master_worker_registrations":{

"help":"Totalnumberofregistrations",

"type":"counter",

"samples":[

{

"value":2

}

]

},

"arangosync_master_worker_storage":{

"help":"Numberoftimesworkerinfoisstored,loaded",

"type":"counter",

"samples":[

{

"value":8,

MultipleDatacenters

331

"labels":{

"kind":"",

"op":"save",

"result":"success"

}

},

{

"value":1,

"labels":{

"kind":"empty",

"op":"load",

"result":"success"

}

}

]

}

...

}

Hint:Togetalistofametricsandtheirhelpinformation,run:

aliasjq='dockerrun--rm-irealguess/jqjq'

curl-sk-u"<user>:<password>"https://<syncmaster-IP>:8629/metrics.json|\

jq'with_entries({key:.key,value:.value.help})'

Security

Firewallsettings

ThecomponentsofArangoSyncuse(TCP)networkconnectionstocommunicatewitheachother.Belowyou'llfindanoverviewoftheseconnectionsandtheTCPportsthatshouldbeaccessible.

1. Thesyncmastersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

ArangoDBagentsandcoordinators(defaultports:8531and8529)Kafkabrokers(defaultport9092)Syncworkers(defaultport8729)

Additionallythesyncmastersmustbeallowedtoconnecttothesyncmastersintheotherdatacenter.

Bydefaultthesyncmasterswilloperateonport8629.

2. Thesyncworkersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

ArangoDBcoordinators(defaultport8529)Kafkabrokers(defaultport9092)Syncmasters(defaultport8629)

Bydefaultthesyncworkerswilloperateonport8729.

AdditionallythesyncworkersmustbeallowedtoconnecttotheKafkabrokersintheotherdatacenter.

3. Kafka

Thekafkabrokersmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

Otherkafkabrokers(defaultport9092)Zookeeper(defaultports2181,2888and3888)

Thedefaultportforkafkais9092.Thedefaultkafkainstallationwillalsoexposesomeprometheusmetricsonport7071.Togainmoreinsightintokafkaopenthisportforyourprometheusinstallation.

4. Zookeeper

Thezookeeperagentsmustbeallowedtoconnecttothefollowingcomponentswithinthesamedatacenter:

OtherzookeeperagentsThesetuphereisabitspecialaszookeeperuses3portsfordifferentoperations.Allagentsneedtobeabletoconnecttoalloftheseports.

MultipleDatacenters

332

BydefaultZookeeperuses:

port2181forclientcommunicationport2888forfollowercommunicationport3888forleaderelections

Certificates

DigitalcertificatesareusedinmanyplacesinArangoSyncforbothencryptionandauthentication.

InArangoSyncallnetworkconnectionsareusingTransportLayerSecurity(TLS),asetofprotocolsthatensurethatallnetworktrafficisencrypted.ForthisTLScertificatesareused.TheserversideofthenetworkconnectionoffersaTLScertificate.Thiscertificateis(often)verifiedbytheclientsideofthenetworkconnection,toensurethatthecertificateissignedbyatrustedCertificateAuthority(CA).Thisensurestheintegrityoftheserver.Inseveralplacesadditionalcertificatesareusedforauthentication.Inthosecasestheclientsideoftheconnectionoffersaclientcertificate(ontopofanexistingTLSconnection).Theserversideoftheconnectionusestheclientcertificatetoauthenticatetheclientand(optionally)decideswhichrightsshouldbeassignedtotheclient.

Note:ArangoSyncdoesallowtheuseofcertificatessignedbyawellknowCA(eg.verisign)howeveritismoreconvenient(andcommon)touseyourownCA.

Formats

Allcertificatesarex509certificateswithapublickey,aprivatekeyandanoptionalchainofcertificatesusedtosignthecertificate(thischainistypicallyprovidedbytheCertificateAuthority(CA)).Dependingontheiruse,certificatesstoredinadifferentformat.

Thefollowingformatsareused:

Publickeyonly(.crt):Afilethatcontainsonlythepublickeyofacertificatewithanoptionalchainofparentcertificates(publickeysofcertificatesusedtosignedthecertificate).Sincethisformatcontainsonlypublickeys,itisnotaproblemifitscontentsareexposed.Itmuststillbestoreitinasafeplacetoavoidlosingit.Privatekeyonly(.key):Afilethatcontainsonlytheprivatekeyofacertificate.Itisvitaltoprotectthesefilesandstoretheminasafeplace.Keyfilewithpublic&privatekey(.keyfile):Afilethatcontainsthepublickeyofacertificate,anoptionalchainofparentcertificatesandaprivatekey.Sincethisformatalsocontainsaprivatekey,itisvitaltoprotectthesefilesandstoretheminasafeplace.Javakeystore(.jks):Afilecontainingasetofpublicandprivatekeys.Itispossibletoprotectaccesstothecontentofthisfileusingakeystorepassword.Sincethisformatcancontainprivatekeys,itisvitaltoprotectthesefilesandstoretheminasafeplace(evenwhenitscontentisprotectedwithakeystorepassword).

Creatingcertificates

ArangoSyncprovidescommandstocreateallcertificatesneeded.

TLSservercertificates

TocreateacertificateusedforTLSserversinthekeyfileformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses).Thenrun:

arangosynccreatetlskeyfile\

--cacert=my-tls-ca.crt--cakey=my-tls-ca.key\

--host=<hostname>\

--keyfile=my-tls-cert.keyfile

Makesuretostorethegeneratedkeyfile(my-tls-cert.keyfile)inasafeplace.

MultipleDatacenters

333

TocreateacertificateusedforTLSserversinthecrt&keyformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses).Thenrun:

arangosynccreatetlscertificate\

--cacert=my-tls-ca.crt--cakey=my-tls-ca.key\

--host=<hostname>\

--cert=my-tls-cert.crt\

--key=my-tls-cert.key\

Makesuretoprotectandstorethegeneratedfiles(my-tls-cert.crt&my-tls-cert.key)inasafeplace.

Clientauthenticationcertificates

Tocreateacertificateusedforclientauthenticationinthekeyfileformat,youneedthepublickeyoftheCA(--cacert),theprivatekeyoftheCA(--cakey)andoneormorehostnames(orIPaddresses)oremailaddresses.Thenrun:

arangosynccreateclient-authkeyfile\

--cacert=my-client-auth-ca.crt--cakey=my-client-auth-ca.key\

[--host=<hostname>|--email=<emailaddress>]\

--keyfile=my-client-auth-cert.keyfile

Makesuretoprotectandstorethegeneratedkeyfile(my-client-auth-cert.keyfile)inasafeplace.

CAcertificates

TocreateaCAcertificateusedtosignTLScertificates,run:

arangosynccreatetlsca\

--cert=my-tls-ca.crt--key=my-tls-ca.key

Makesuretoprotectandstorebothgeneratedfiles(my-tls-ca.crt&my-tls-ca.key)inasafeplace.Note:CAcertificateshaveamuchlongerlifetimethannormalcertificates.Thereforeevenmorecareisneededtostorethemsafely.

TocreateaCAcertificateusedtosignclientauthenticationcertificates,run:

arangosynccreateclient-authca\

--cert=my-client-auth-ca.crt--key=my-client-auth-ca.key

Makesuretoprotectandstorebothgeneratedfiles(my-client-auth-ca.crt&my-client-auth-ca.key)inasafeplace.Note:CAcertificateshaveamuchlongerlifetimethannormalcertificates.Thereforeevenmorecareisneededtostorethemsafely.

Renewingcertificates

Allcertificateshavemetainformationinthemthelimittheiruseinfunction,target&lifetime.Acertificatecreatedforclientauthentication(function)cannotbeusedasaTLSservercertificate(sameistrueforthereverse).Acertificateforhostmyserver(target)cannotbeusedforhostanotherserver.AcertficiatethatisvaliduntilOctober2017(limetime)cannotbeusedafterOctober2017.

Ifanythingchangesinfunction,targetorlifetimeyouneedanewcertificate.

Theprocedureforcreatingarenewedcertificateisthesameasforcreatinga"first"certificate.Aftercreatingtherenewedcertificatetheprocess(es)usingthemhavetobeupdated.Thismeanrestartingthem.AllArangoSynccomponentsaredesignedtosupportstoppingandstartingsingleinstances,butdonotrestartmorethan1instanceatthesametime.Assoonas1instancehasbeenrestarted,giveitsometimeto"catchup"beforerestartingthenextinstance.

MultipleDatacenters

334

ArangoDBKubernetesOperatorTheArangoDBKubernetesOperator(kube-arangodb)isasetofoperatorsthatyoudeployinyourKubernetesclusterto:

ManagedeploymentsoftheArangoDBdatabaseProvidePersistentVolumesonlocalstorageofyournodesforoptimalstorageperformance.ConfigureArangoDBDatacentertoDatacenterreplication

Eachoftheseusesinvolvesadifferentcustomresource.

UseanArangoDeploymentresourcetocreateanArangoDBdatabasedeployment.UseanArangoLocalStorageresourcetoprovidelocalPersistentVolumesforoptimalI/Operformance.UseanArangoDeploymentReplicationresourcetoconfigureArangoDBDatacentertoDatacenterreplication.

ContinuewithUsingtheArangoDBKubernetesOperatortolearnhowtoinstalltheArangoDBKubernetesoperatorandcreateyourfirstdeployment.

Kubernetes

335

UsingtheArangoDBKubernetesOperator

Installation

TheArangoDBKubernetesOperatorneedstobeinstalledinyourKubernetesclusterfirst.

Todoso,run(replace<version>withtheversionoftheoperatorthatyouwanttoinstall):

exportURLPREFIX=https://raw.githubusercontent.com/arangodb/kube-arangodb/<version>/manifests

kubectlapply-f$URLPREFIX/crd.yaml

kubectlapply-f$URLPREFIX/arango-deployment.yaml

TouseArangoLocalStorageresources,alsorun:

kubectlapply-f$URLPREFIX/arango-storage.yaml

TouseArangoDeploymentReplicationresources,alsorun:

kubectlapply-f$URLPREFIX/arango-deployment-replication.yaml

YoucanfindthelatestreleaseoftheArangoDBKubernetesOperatorinthekube-arangodbrepository.

ArangoDBdeploymentcreationOncetheoperatorisrunning,youcancreateyourArangoDBdatabasedeploymentbycreatingaArangoDeploymentcustomresourceanddeployingitintoyourKubernetescluster.

Forexample(allexamplescanbefoundinthekube-arangodbrepository):

kubectlapply-fexamples/simple-cluster.yaml

Deploymentremoval

ToremoveanexistingArangoDBdeployment,deletethecustomresource.Theoperatorwillthendeleteallcreatedresources.

Forexample:

kubectldelete-fexamples/simple-cluster.yaml

NotethatthiswillalsodeletealldatainyourArangoDBdeployment!

Ifyouwanttokeepyourdata,makesuretocreateabackupbeforeremovingthedeployment.

Operatorremoval

ToremovetheentireArangoDBKubernetesOperator,removeallclustersfirstandthenremovetheoperatorbyrunning:

kubectldeletedeploymentarango-deployment-operator

#IfÀrangoLocalStorageòperatorisinstalled

kubectldeletedeployment-nkube-systemarango-storage-operator

#IfÀrangoDeploymentReplicationòperatorisinstalled

kubectldeletedeploymentarango-deployment-replication-operator

UsingtheOperator

336

https://github.com/arangodb/kube-arangodb/releases/latest

https://github.com/arangodb/kube-arangodb/tree/master/examples

Seealso

DriverconfigurationScalingUpgrading

UsingtheOperator

337

ArangoDeploymentCustomResourceTheArangoDBDeploymentOperatorcreatesandmaintainsArangoDBdeploymentsinaKubernetescluster,givenadeploymentspecification.ThisdeploymentspecificationisaCustomResourcefollowingaCustomResourceDefinitioncreatedbytheoperator.

ExampleminimaldeploymentdefinitionofanArangoDBdatabasecluster:



metadata:

name:"example-arangodb-cluster"

spec:

mode:Cluster

Examplemoreelaboratedeploymentdefinition:



metadata:


spec:

mode:Cluster

environment:Production

agents:

count:3

args:

---log.level=debug

resources:

requests:

storage:8Gi

storageClassName:ssd

dbservers:

count:5

resources:

requests:

storage:80Gi

storageClassName:ssd

coordinators:

count:3

image:"arangodb/arangodb:3.3.4"

Specificationreference

Belowyou'llfindallsettingsoftheArangoDeploymentcustomresource.Severalsettingsareforvariousgroupsofservers.Theseareindicatedwith<group>where<group>canbeanyof:

agentsforallagentsofaClusterorActiveFailoverpair.dbserversforalldbserversofaCluster.coordinatorsforallcoordinatorsofaCluster.singleforallsingleserversofaSingleinstanceorActiveFailoverpair.syncmastersforallsyncmastersofaCluster.syncworkersforallsyncworkersofaCluster.

spec.mode:string

Thissettingspecifiesthetypeofdeploymentyouwanttocreate.Possiblevaluesare:

Cluster(default)Fullcluster.Defaultsto3agents,3dbservers&3coordinators.ActiveFailoverActive-failoversinglepair.Defaultsto3agentsand2singleservers.SingleSingleserveronly(notethisdoesnotprovidehighavailabilityorreliability).

Thissettingcannotbechangedafterthedeploymenthasbeencreated.


338

spec.environment:string

Thissettingspecifiesthetypeofenvironmentinwhichthedeploymentiscreated.Possiblevaluesare:

Development(default)Thisvalueoptimizesthedeploymentfordevelopmentuse.Itispossibletorunadeploymentonasmallnumberofnodes(e.g.minikube).ProductionThisvalueoptimizesthedeploymentforproductionuse.Itputsrequiredaffinityconstraintsonallpodstoavoidagents&dbserversfromrunningonthesamemachine.

spec.image:string

ThissettingspecifiesthedockerimagetouseforallArangoDBservers.Inadevelopmentenvironmentthissettingdefaultstoarangodb/arangodb:latest.Forproductionenvironmentsthisisarequiredsettingwithoutadefaultvalue.Itishighlyrecommendtouseexplicitversion(notlatest)forproductionenvironments.

spec.imagePullPolicy:string

ThissettingspecifiesthepullpolicyforthedockerimagetouseforallArangoDBservers.Possiblevaluesare:

IfNotPresent(default)topullonlywhentheimageisnotfoundonthenode.Alwaystoalwayspulltheimagebeforeusingit.

spec.storageEngine:string

Thissettingspecifiesthetypeofstorageengineusedforallserversinthecluster.Possiblevaluesare:

MMFilesTousetheMMFilesstorageengine.RocksDB(default)TousetheRocksDBstorageengine.

Thissettingcannotbechangedaftertheclusterhasbeencreated.

spec.downtimeAllowed:bool

ThissettingisusedtoallowautomaticreconciliationactionsthatyieldsomedowntimeoftheArangoDBdeployment.Whenthissettingissettofalse(thedefault),noautomaticactionthatmayresultindowntimeisallowed.Iftheneedforsuchanactionisdetected,aneventisaddedtotheArangoDeployment.

Oncethissettingissettotrue,theautomaticactionisexecuted.

Operationsthatmayresultindowntimeare:

RotatingTLSCAcertificate

Note:ItisstillpossiblethatthereissomedowntimewhentheKubernetesclusterisdown,orinabadstate,irrespectiveofthevalueofthissetting.

spec.rocksdb.encryption.keySecretName

ThissettingspecifiesthenameofakubernetesSecretthatcontainsanencryptionkeyusedforencryptingalldatastoredbyArangoDBservers.Whenanencryptionkeyisused,encryptionofthedataintheclusterisenabled,withoutitencryptionisdisabled.Thedefaultvalueisempty.

ThisrequirestheEnterpriseversion.

Theencryptionkeycannotbechangedaftertheclusterhasbeencreated.

Thesecretspecifiedbythissetting,musthaveadatafieldnamed'key'containinganencryptionkeythatisexactly32byteslong.

spec.externalAccess.type:string

ThissettingspecifiesthetypeofServicethatwillbecreatedtoprovideaccesstotheArangoDBdeploymentfromoutsidetheKubernetescluster.Possiblevaluesare:


339

NoneTolimitaccesstoapplicationrunninginsidetheKubernetescluster.LoadBalancerTocreateaServiceoftypeLoadBalancerfortheArangoDBdeployment.NodePortTocreateaServiceoftypeNodePortfortheArangoDBdeployment.Auto(default)TocreateaServiceoftypeLoadBalancerandfallbacktoaServiceortypeNodePortwhentheLoadBalancerisnotassignedanIPaddress.

spec.externalAccess.loadBalancerIP:string

ThissettingspecifiestheIPusedtofortheLoadBalancertoexposetheArangoDBdeploymenton.Thissettingisusedwhenspec.externalAccess.typeissettoLoadBalancerorAuto.

Ifyoudonotspecifythissetting,anIPwillbechosenautomaticallybytheload-balancerprovisioner.

spec.externalAccess.nodePort:int

ThissettingspecifiestheportusedtoexposetheArangoDBdeploymenton.Thissettingisusedwhenspec.externalAccess.typeissettoNodePortorAuto.

Ifyoudonotspecifythissetting,arandomportwillbechosenautomatically.

spec.auth.jwtSecretName:string

ThissettingspecifiesthenameofakubernetesSecretthatcontainstheJWTtokenusedforaccessingallArangoDBservers.Whennonameisspecified,itdefaultsto<deployment-name>-jwt.Todisableauthentication,setthisvaluetoNone.

IfyouspecifyanameofaSecret,thatsecretmusthavethetokeninadatafieldnamedtoken.

IfyouspecifyanameofaSecretthatdoesnotexist,arandomtokeniscreatedandstoredinaSecretwithgivenname.

ChangingaJWTtokenresultsinstoppingtheentireclusterandrestartingit.

spec.tls.caSecretName:string

ThissettingspecifiesthenameofakubernetesSecretthatcontainsastandardCAcertificate+privatekeyusedtosigncertificatesforindividualArangoDBservers.Whennonameisspecified,itdefaultsto<deployment-name>-ca.Todisableauthentication,setthisvaluetoNone.

IfyouspecifyanameofaSecretthatdoesnotexist,aself-signedCAcertificate+keyiscreatedandstoredinaSecretwithgivenname.

ThespecifiedSecret,mustcontainthefollowingdatafields:

ca.crtPEMencodedpublickeyoftheCAcertificateca.keyPEMencodedprivatekeyoftheCAcertificate

spec.tls.altNames:[]string

Thissettingspecifiesalistofalternatenamesthatwillbeaddedtoallgeneratedcertificates.ThesenamescanbeDNSnamesoremailaddresses.Thedefaultvalueisempty.

spec.tls.ttl:duration

Thissettingspecifiesthetimetoliveofallgeneratedservercertificates.Thedefaultvalueis2160h(about3month).

Whentheservercertificateisabouttoexpire,itwillbeautomaticallyreplacedbyanewoneandtheaffectedserverwillberestarted.

Note:ThetimetoliveoftheCAcertificate(whencreatedautomatically)willbesetto10years.

spec.sync.enabled:bool

Thissettingenables/disablessupportfordatacenter2datacenterreplicationinthecluster.Whenenabled,theclusterwillcontainanumberofsyncmaster&syncworkerservers.Thedefaultvalueisfalse.


340

spec.sync.externalAccess.type:string

ThissettingspecifiesthetypeofServicethatwillbecreatedtoprovideaccesstotheArangoSyncsyncMastersfromoutsidetheKubernetescluster.Possiblevaluesare:

NoneTolimitaccesstoapplicationsrunninginsidetheKubernetescluster.LoadBalancerTocreateaServiceoftypeLoadBalancerfortheArangoSyncSyncMasters.NodePortTocreateaServiceoftypeNodePortfortheArangoSyncSyncMasters.Auto(default)TocreateaServiceoftypeLoadBalancerandfallbacktoaServiceortypeNodePortwhentheLoadBalancerisnotassignedanIPaddress.

NotethatwhenyouspecifyavalueofNone,aServicewillstillbecreated,butoftypeClusterIP.

spec.sync.externalAccess.loadBalancerIP:string

ThissettingspecifiestheIPusedfortheLoadBalancertoexposetheArangoSyncSyncMasterson.Thissettingisusedwhenspec.sync.externalAccess.typeissettoLoadBalancerorAuto.

Ifyoudonotspecifythissetting,anIPwillbechosenautomaticallybytheload-balancerprovisioner.

spec.sync.externalAccess.nodePort:int

ThissettingspecifiestheportusedtoexposetheArangoSyncSyncMasterson.Thissettingisusedwhenspec.sync.externalAccess.typeissettoNodePortorAuto.

Ifyoudonotspecifythissetting,arandomportwillbechosenautomatically.

spec.sync.externalAccess.masterEndpoint:[]string

Thissettingspecifiesthemasterendpoint(s)advertisedbytheArangoSyncSyncMasters.Ifnotset,thissettingdefaultsto:

Ifspec.sync.externalAccess.loadBalancerIPisset,itdefaultstohttps://<load-balancer-ip>:<8629>.Otherwiseitdefaultstohttps://<sync-service-dns-name>:<8629>.

spec.sync.externalAccess.accessPackageSecretNames:[]string

ThissettingspecifiesthenamesofzeroofmoreSecretsthatwillbecreatedbythedeploymentoperatorcontaining"accesspackages".AnaccesspackagecontainsthoseSecretsthatareneededtoaccesstheSyncMastersofthisArangoDeployment.

Byremovinganamefromthissetting,thecorrespondingSecretisalsodeleted.Notethattoremoveallaccesspackages,leaveanemptyarrayinplace([]).Completelyremovingthesettingresultsinnotmodifyingthelist.

SeetheArangoDeploymentReplicationspecificationformoreinformationonaccesspackages.

spec.sync.auth.jwtSecretName:string

ThissettingspecifiesthenameofakubernetesSecretthatcontainstheJWTtokenusedforaccessingallArangoSyncmasterservers.Whennotspecified,thespec.auth.jwtSecretNamevalueisused.

IfyouspecifyanameofaSecretthatdoesnotexist,arandomtokeniscreatedandstoredinaSecretwithgivenname.

spec.sync.auth.clientCASecretName:string

ThissettingspecifiesthenameofakubernetesSecretthatcontainsaPEMencodedCAcertificateusedforclientcertificateverificationinallArangoSyncmasterservers.Thisisarequiredsettingwhenspec.sync.enabledistrue.Thedefaultvalueisempty.

spec.sync.mq.type:string

ThissettingsetsthetypeofmessagequeueusedbyArangoSync.Possiblevaluesare:

Direct(default)fordirectHTTPconnectionsbetweenthe2datacenters.


341

spec.sync.tls.caSecretName:string

ThissettingspecifiesthenameofakubernetesSecretthatcontainsastandardCAcertificate+privatekeyusedtosigncertificatesforindividualArangoSyncmasterservers.

Whennonameisspecified,itdefaultsto<deployment-name>-sync-ca.

IfyouspecifyanameofaSecretthatdoesnotexist,aself-signedCAcertificate+keyiscreatedandstoredinaSecretwithgivenname.

ThespecifiedSecret,mustcontainthefollowingdatafields:

ca.crtPEMencodedpublickeyoftheCAcertificateca.keyPEMencodedprivatekeyoftheCAcertificate

spec.sync.tls.altNames:[]string

Thissettingspecifiesalistofalternatenamesthatwillbeaddedtoallgeneratedcertificates.ThesenamescanbeDNSnamesoremailaddresses.Thedefaultvalueisempty.

spec.sync.monitoring.tokenSecretName:string

ThissettingspecifiesthenameofakubernetesSecretthatcontainsthebearertokenusedforaccessingallmonitoringendpointsofallArangoSyncservers.Whennotspecified,nomonitoringtokenisused.Thedefaultvalueisempty.

spec.ipv6.forbidden:bool

ThissettingpreventstheuseofIPv6addressesbyArangoDBservers.Thedefaultisfalse.

spec.<group>.count:number

Thissettingspecifiesthenumberofserverstostartforthegivengroup.Fortheagentgroup,thisvaluemustbeapositive,oddnumber.Thedefaultvalueis3forallgroupsexceptsingle(therethedefaultis1forspec.mode:Singleand2forspec.mode:ActiveFailover).

Forthesyncworkersgroup,itishighlyrecommendedtousethesamenumberasforthedbserversgroup.

spec.<group>.args:[string]

Thissettingspecifiesadditionalcommandlineargumentspassedtoallserversofthisgroup.Thedefaultvalueisanemptyarray.

spec.<group>.resources.requests.cpu:cpuUnit

ThissettingspecifiestheamountofCPUrequestedbyserverofthisgroup.

Seehttps://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/fordetails.

spec.<group>.resources.requests.memory:memoryUnit

Thissettingspecifiestheamountofmemoryrequestedbyserverofthisgroup.

Seehttps://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/fordetails.

spec.<group>.resources.requests.storage:storageUnit

Thissettingspecifiestheamountofstoragerequiredforeachserverofthisgroup.Thedefaultvalueis8Gi.

Thissettingisnotavailableforgroupcoordinators,syncmasters&syncworkersbecauseserversinthesegroupsdonotneedpersistentstorage.

spec.<group>.serviceAccountName:string


342

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/

ThissettingspecifiestheserviceAccountNameforthePodscreatedforeachserverofthisgroup.

UsinganalternativeServiceAccountistypicallyusedtoseparateaccessrights.TheArangoDBdeploymentsdonotrequireanyspecialrights.

spec.<group>.storageClassName:string

ThissettingspecifiesthestorageClassforthePersistentVolumescreatedforeachserverofthisgroup.

Thissettingisnotavailableforgroupcoordinators,syncmasters&syncworkersbecauseserversinthesegroupsdonotneedpersistentstorage.

spec.<group>.tolerations:[Toleration]

ThissettingspecifiesthetolerationsforthePodscreatedforeachserverofthisgroup.

Bydefault,suitabletolerationsaresetforthefollowingkeyswiththeNoExecuteeffect:

node.kubernetes.io/not-ready

node.kubernetes.io/unreachable

node.alpha.kubernetes.io/unreachable(willberemovedinfutureversion)

Formoreinformationontolerations,consulttheKubernetesdocumentation.


343

https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

ConfiguringyourdriverforArangoDBaccessInthischapteryou'lllearnhowtoconfigureadriverforaccessinganArangoDBdeploymentinKubernetes.

Theexactmethodstoconfigureadriverarespecifictothatdriver.

Databaseendpoint(s)

Theendpoint(s)(orURLs)tocommunicatewithisthemostimportantparameteryourneedtoconfigureinyourdriver.

FindingtherightendpointsdependonwetheryourclientapplicationisrunninginthesameKubernetesclusterastheArangoDBdeploymentornot.

ClientapplicationinsameKubernetescluster

IfyourclientapplicationisrunninginthesameKubernetesclusterastheArangoDBdeployment,youshouldconfigureyourdrivertousethefollowingendpoint:

https://<deployment-name>.<namespace>.svc:8529

Onlyifyourdeploymenthassetspec.tls.caSecretNametoNone,shouldyouusehttpinsteadofhttps.

ClientapplicationoutsideKubernetescluster

IfyourclientapplicationisrunningoutsidetheKubernetesclusterinwhichtheArangoDBdeploymentisrunning,yourdriverendpointdependsontheexternal-accessconfigurationofyourArangoDBdeployment.

Iftheexternal-accessoftheArangoDBdeploymentisoftypeLoadBalancer,thenusetheIPaddressofthatLoadBalancerlikethis:

https://<load-balancer-ip>:8529

Iftheexternal-accessoftheArangoDBdeploymentisoftypeNodePort,thenusetheIPaddress(es)oftheNodesoftheKubernetescluster,combinedwiththeNodePortthatisusedbytheexternal-accessservice.

Forexample:

https://<kubernetes-node-1-ip>:30123

Youcanfindthetypeofexternal-accessbyinspectingtheexternal-accessService.Todoso,runthefollowingcommand:

kubectlgetservice-n<namespace-of-deployment><deployment-name>-ea

Theoutputlookslikethis:

NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGESELECTOR

example-simple-cluster-eaLoadBalancer10.106.175.38192.168.10.2088529:31890/TCP1sapp=arangodb,arango_de

ployment=example-simple-cluster,role=coordinator

Inthiscasetheexternal-accessisoftypeLoadBalancerwithaload-balancerIPaddressof192.168.10.208.Thisresultsinanendpointofhttps://192.168.10.208:8529.

TLSsettingsAsmentionedbeforetheArangoDBdeploymentmanagedbytheArangoDBoperatorwilluseasecure(TLS)connectionunlessyousetspec.tls.caSecretNametoNoneinyourArangoDeployment.

DriverConfiguration

344

Whenusingasecureconnection,youcanchoosetoverifytheservercertificatesprovidesbytheArangoDBserversornot.

Ifyouwanttoverifythesecertificates,configureyourdriverwiththeCAcertificatefoundinaKubernetesSecretfoundinthesamenamespaceastheArangoDeployment.

ThenameofthisSecretisstoredinthespec.tls.caSecretNamesettingoftheArangoDeployment.Ifyoudon'tsetthissettingexplicitly,itwillbesetautomatically.

ThenfetchtheCAsecretusingthefollowingcommand(oruseaKubernetesclientlibrarytofetchit):

kubectlgetsecret-n<namespace><secret-name>--template='{{index.data"ca.crt"}}'|base64-D>ca.crt

Thisresultsinafilecalledca.crtcontainingaPEMencoded,x509CAcertificate.

QueryrequestsFormostclientrequestsmadebyadriver,itdoesnotmatterifthereisanykindofload-balancerbetweenyourclientapplicationandtheArangoDBdeployment.

NotethatevenasimpleServiceoftypeClusterIPalreadybehavesasaload-balancer.

TheexceptiontothisiscursorrelatedrequestsmadetoanArangoDBClusterdeployment.Thecoordinatorthathandlesaninitialqueryrequest(thatresultsinaCursor)willsavesomein-memorystateinthatcoordinator,iftheresultofthequeryistoobigtobetransferbackintheresponseoftheinitialrequest.

Follow-uprequestshavetobemadetofetchtheremainingdata.Thesefollow-uprequestsmustbehandledbythesamecoordinatortowhichtheinitialrequestwasmade.

Assoonasthereisaload-balancerbetweenyourclientapplicationandtheArangoDBcluster,itisuncertainwhichcoordinatorwillactuallyhandlethefollow-uprequest.

Toresolvethisuncertainty,makesuretorunyourclientapplicationinthesameKubernetesclusterandsynchronizeyourendpointsbeforemakingtheinitialqueryrequest.Thiswillresultintheuse(bythedriver)ofinternalDNSnamesofallcoordinators.Afollow-uprequestcanthenbesenttoexactlythesamecoordinator.

IfyourclientapplicationisrunningoutsidetheKubernetesclusterthisismuchhardertosolve.Theeasiestwaytoworkaroundit,isbymakingsurethatthequeryresultsaresmallenough.Whenthatisnotfeasible,itisalsopossibletoresolvethiswhentheinternalDNSnamesofyourKubernetesclusterareexposedtoyourclientapplicationandtheresulingIPaddressesarerouteablefromyourclientapplication.ToexposeinternalDNSnamesofyourKubernetescluster,yourcanuseCoreDNS.

DriverConfiguration

345

https://coredns.io

AuthenticationTheArangoDBKubernetesOperatorwillbydefaultcreateArangoDBdeploymentsthatrequireauthenticationtoaccessthedatabase.

ItusesasingleJWTsecret(storedinaKubernetessecret)toprovidesuper-useraccessbetweenallserversofthedeploymentaswellasaccessfromtheArangoDBOperatortothedeployment.

Todisableauthentication,setspec.auth.jwtSecretNametoNone.

Initiallythedeploymentisaccessiblethroughthewebuser-interfaceandAPI's,usingtheuserrootwithanemptypassword.Makesuretochangethispasswordimmediatelyafterstartingthedeployment!

Seealso

Secureconnections(TLS)

Authentication

346

ScalingTheArangoDBKubernetesOperatorsupportsupanddownscalingofthenumberofdbservers&coordinators.

Currentlyitisnotpossibletochangethenumberofagentsofacluster.

Thescaleupordown,changethenumberofserversinthecustomresource.

E.g.changespec.dbservers.countfrom3to4.

Thenapplytheupdatedresourceusing:

kubectlapply-fyourCustomResourceFile.yaml

Inspectthestatusofthecustomresourcetomonitortheprogressofthescalingoperation.

Scaling

347

UpgradingTheArangoDBKubernetesOperatorsupportsupgradinganArangoDBfromoneversiontothenext.

UpgradeanArangoDBdeployment

Toupgradeacluster,changetheversionbychangingthespec.imagesettingandtheapplytheupdatedcustomresourceusing:

kubectlapply-fyourCustomResourceFile.yaml

TheArangoDBoperatorwillperformansequentialupgradeofallserversinyourdeployment.Onlyoneserverisupgradedatatime.

Forpatchlevelupgrades(e.g.3.3.9to3.3.10)eachserverisstoppedandrestartedwiththenewversion.

Forminorlevelupgrades(e.g.3.3.9to3.4.0)eachserverisstopped,thenthenewversionisstartedwith--database.auto-upgradeandoncethatisfinishthenewversionisstartedwiththenormalarguments.

Theprocessformajorlevelupgradesdependsonthespecificversion.

UpgradetheoperatoritselfToupdatetheArangoDBKubernetesOperatoritselftoanewversion,updatetheimageversionofthedeploymentresourceandapplyitusing:

kubectlapply-fexamples/yourUpdatedDeployment.yaml

Seealso

Scaling

Upgrading

348

Configuration&secretsAnArangoDBclusterhaslotsofconfigurationoptions.SomewillbesupporteddirectlyintheArangoDBOperator,otherswillhavetospecifiedseparately.

Built-inoptions

Allbuilt-inoptionsarepassedtoArangoDBserversviacommandlineargumentsconfiguredinthePod-spec.

Otherconfigurationoptions

Allcommandlineoptionsofarangod(andarangosync)areavailablebyaddingoptionstothespec.<group>.argslistofagroupofservers.

Theseargumentsareaddedtothcommandlinecreatedfortheseservers.

Secrets

TheArangoDBclusterneedsseveralsecretssuchasJWTtokensTLScertificatesandsoon.

AllthesesecretsarestoredasKubernetesSecretsandpassedtotheapplicablePodsasfiles,mappedintothePodsfilesystem.

Thenameofthesecretisspecifiedinthecustomresource.Forexample:

apiVersion:"cluster.arangodb.com/v1alpha"

kind:"Cluster"

metadata:


spec:

mode:Cluster

auth:

jwtSecretName:<name-of-JWT-token-secret>

ArangoDBConfiguration&Secrets

349

MetricsTheArangoDBKubernetesOperator(kube-arangodb)exposesmetricsofitsoperationsinaformatthatiscompatiblewithPrometheus.

ThemetricsareexposedthroughHTTPSonport8528underpath/metrics.

Lookatexamples/metricsforexamplesofServicesandServiceMonitorsyoucanusetointegratewithPrometheusthroughthePrometheus-OperatorbyCoreOS.

Metrics

350

https://prometheus.io

https://github.com/arangodb/kube-arangodb/tree/master/examples/metrics

https://github.com/coreos/prometheus-operator

ServicesandloadbalancerTheArangoDBKubernetesOperatorwillcreateservicesthatcanbeusedtoreachtheArangoDBserversfrominsidetheKubernetescluster.

Bydefault,theArangoDBKubernetesOperatorwillalsocreateanadditionalservicetoreachtheArangoDBdeploymentfromoutsidetheKubernetescluster.

ForexposingtheArangoDBdeploymenttotheoutside,thereare2options:

UsingaNodePortservice.Thiswillexposethedeploymentonaspecificport(above30.000)onallnodesoftheKubernetescluster.UsingaLoadBalancerservice.Thiswillexposethedeploymentonaload-balancerthatisprovisionedbytheKubernetescluster.

TheLoadBalanceroptionisthemostconvenient,butnotallKubernetesclustersareabletoprovisionaload-balancer.Thereforeweofferathird(anddefault)option:Auto.Inthisoption,theArangoDBKubernetesOperatortriestocreateaLoadBalancerservice.ItthenwaitsforuptoaminutefortheKubernetesclustertoprovisionaload-balancerforit.Ifthathasnothappenedafteraminute,theserviceisreplacedbyaserviceoftypeNodePort.

Toinspectthecreatedservice,run:

kubectlgetservices<deployment-name>-ea

TousetheArangoDBserversfromoutsidetheKubernetesclusteryouhavetoaddanotherserviceasexplainedbelow.

ServicesIfyoudonotwanttheArangoDBKubernetesOperatortocreateanexternal-accessserviceforyou,setspec.externalAccess.TypetoNone.

Ifyouwanttocreateexternalaccessservicesmanually,followtheinstructionsbelow.

Singleserver

Forasingleserverdeployment,theoperatorcreatesasingleServicenamed<deployment-name>.ThisservicehasanormalclusterIPaddress.

Fullcluster

Forafullclusterdeployment,theoperatorcreatestwoServices.

<deployment-name>-intaheadlessServiceintendedtoprovideDNSnamesforallpodscreatedbytheoperator.ItselectsallArangoDB&ArangoSyncserversinthecluster.

<deployment-name>anormalServicethatselectsonlythecoordinatorsofthecluster.ThisServiceisconfiguredwithClientIPsessionaffinity.Thisisneededforcursorrequests,sincetheyareboundtoaspecificcoordinator.

Whenthecoordinatorsareaskedtoprovideendpointsofthecluster(e.g.whencallingclient.SynchronizeEndpoints()inthegodriver)theDNSnamesoftheindividualPodswillbereturned(<pod>.<deployment-name>-int.<namespace>.svc)

FullclusterwithDC2DC

Forafullclusterwithdatacenterreplicationdeployment,thesameServicesarecreatedasforaFullcluster,withthefollowingadditions:

<deployment-name>-syncanormalServicethatselectsonlythesyncmastersofthecluster.

Loadbalancer


351

IfyouwantfullcontroloftheServicesneededtoaccesstheArangoDBdeploymentfromoutsideyourKubernetescluster,setspec.externalAccess.TypeoftheArangoDeploymenttoNoneandcreateaServiceasspecifiedbelow.

CreateaServiceoftypeLoadBalancerorNodePort,dependingonyourKubernetesdeployment.

Thisserviceshouldselect:

arango_deployment:<deployment-name>

role:coordinator

ThefollowingexampleyieldsaserviceoftypeLoadBalancerwithaspecificloadbalancerIPaddress.Withthisservice,theArangoDBclustercannowbereachedonhttps://1.2.3.4:8529.

kind:Service

apiVersion:v1

metadata:

name:arangodb-cluster-exposed

spec:

selector:

arango_deployment:arangodb-cluster

role:coordinator

type:LoadBalancer

loadBalancerIP:1.2.3.4

ports:

-protocol:TCP

port:8529

targetPort:8529

ThefollowingexampleyieldsaserviceoftypeNodePortwiththeArangoDBclusterexposedonport30529ofallnodesoftheKubernetescluster.

kind:Service

apiVersion:v1

metadata:

name:arangodb-cluster-exposed

spec:

selector:

arango_deployment:arangodb-cluster

role:coordinator

type:NodePort

ports:

-protocol:TCP

port:8529

targetPort:8529

nodePort:30529


352

ArangoDeploymentReplicationCustomResourceTheArangoDBReplicationOperatorcreatesandmaintainsArangoDBarangosyncconfigurationsinaKubernetescluster,givenareplicationspecification.ThisreplicationspecificationisaCustomResourcefollowingaCustomResourceDefinitioncreatedbytheoperator.

Exampleminimalreplicationdefinitionfor2ArangoDBclusterwithsyncinthesameKubernetescluster:



metadata:

name:"replication-from-a-to-b"

spec:

source:

deploymentName:cluster-a

auth:

keyfileSecretName:cluster-a-sync-auth

destination:

deploymentName:cluster-b

Thisdefinitionresultsin:

thearangosyncSyncMasterindeploymentcluster-biscalledtoconfigureasynchronizationfromthesyncmastersincluster-atothesyncmastersincluster-b,usingtheclientauthenticationcertificatestoredinSecretcluster-a-sync-auth.Toaccesscluster-a,theJWTsecretfoundinthedeploymentofcluster-aisused.Toaccesscluster-b,theJWTsecretfoundinthedeploymentofcluster-bisused.

ExamplereplicationdefinitionforreplicatingfromasourcethatisoutsidethecurrentKubernetesclustertoadestinationthatisinthesameKubernetescluster:



metadata:

name:"replication-from-a-to-b"

spec:

source:

masterEndpoint:["https://163.172.149.229:31888","https://51.15.225.110:31888","https://51.15.229.133:31888"]

auth:

keyfileSecretName:cluster-a-sync-auth

tls:

caSecretName:cluster-a-sync-ca

destination:

deploymentName:cluster-b


thearangosyncSyncMasterindeploymentcluster-biscalledtoconfigureasynchronizationfromthesyncmasterslocatedatthegivenlistofendpointURL'stothesyncmasterscluster-b,usingtheclientauthenticationcertificatestoredinSecretcluster-a-sync-auth.Toaccesscluster-a,thekeyfile(containingaclientauthenticationcertificate)isused.Toaccesscluster-b,theJWTsecretfoundinthedeploymentofcluster-bisused.

Specificationreference

Belowyou'llfindallsettingsoftheArangoDeploymentReplicationcustomresource.

spec.source.deploymentName:string

ThissettingspecifiesthenameofanArangoDeploymentresourcethatrunsaclusterwithsyncenabled.

Thisclusterconfiguredasthereplicationsource.


353

spec.source.masterEndpoint:[]string

ThissettingspecifieszeroormoremasterendpointURL'softhesourcecluster.

UsethissettingifthesourceclusterisnotrunninginsideaKubernetesclusterthatisreachablefromtheKubernetesclustertheArangoDeploymentReplicationresourceisdeployedin.

Specifyingthissettingandspec.source.deploymentNameatthesametimeisnotallowed.

spec.source.auth.keyfileSecretName:string

ThissettingspecifiesthenameofaSecretcontainingaclientauthenticationcertificatecalledtls.keyfileusedtoauthenticatewiththeSyncMasteratthespecifiedsource.

Ifspec.source.auth.userSecretNamehasnotbeenset,theclientauthenticationcertificatefoundinthesecretwiththisnameisalsousedtoconfigurethesynchronizationandfetchthesynchronizationstatus.

Thissettingisrequired.

spec.source.auth.userSecretName:string

ThissettingspecifiesthenameofaSecretcontainingausername&passwordusedtoauthenticatewiththeSyncMasteratthespecifiedsourceinordertoconfiguresynchronizationandfetchsynchronizationstatus.

Theuseridentifiedbytheusernamemusthavewriteaccessinthe_systemdatabaseofthesourceArangoDBcluster.

spec.source.tls.caSecretName:string

ThissettingspecifiesthenameofaSecretcontainingaTLSCAcertificateca.crtusedtoverifytheTLSconnectioncreatedbytheSyncMasteratthespecifiedsource.

Thissettingisrequired,unlessspec.source.deploymentNamehasbeenset.

spec.destination.deploymentName:string

ThissettingspecifiesthenameofanArangoDeploymentresourcethatrunsaclusterwithsyncenabled.

Thisclusterconfiguredasthereplicationdestination.

spec.destination.masterEndpoint:[]string

ThissettingspecifieszeroormoremasterendpointURL'softhedestinationcluster.

UsethissettingifthedestinationclusterisnotrunninginsideaKubernetesclusterthatisreachablefromtheKubernetesclustertheArangoDeploymentReplicationresourceisdeployedin.

Specifyingthissettingandspec.destination.deploymentNameatthesametimeisnotallowed.

spec.destination.auth.keyfileSecretName:string

ThissettingspecifiesthenameofaSecretcontainingaclientauthenticationcertificatecalledtls.keyfileusedtoauthenticatewiththeSyncMasteratthespecifieddestination.

Ifspec.destination.auth.userSecretNamehasnotbeenset,theclientauthenticationcertificatefoundinthesecretwiththisnameisalsousedtoconfigurethesynchronizationandfetchthesynchronizationstatus.

Thissettingisrequired,unlessspec.destination.deploymentNameorspec.destination.auth.userSecretNamehasbeenset.

Specifyingthissettingandspec.destination.userSecretNameatthesametimeisnotallowed.

spec.destination.auth.userSecretName:string


354

ThissettingspecifiesthenameofaSecretcontainingausername&passwordusedtoauthenticatewiththeSyncMasteratthespecifieddestinationinordertoconfiguresynchronizationandfetchsynchronizationstatus.

Theuseridentifiedbytheusernamemusthavewriteaccessinthe_systemdatabaseofthedestinationArangoDBcluster.

Specifyingthissettingandspec.destination.keyfileSecretNameatthesametimeisnotallowed.

spec.destination.tls.caSecretName:string

ThissettingspecifiesthenameofaSecretcontainingaTLSCAcertificateca.crtusedtoverifytheTLSconnectioncreatedbytheSyncMasteratthespecifieddestination.

Thissettingisrequired,unlessspec.destination.deploymentNamehasbeenset.

Authenticationdetails

TheauthenticationsettingsinaArangoDeploymentReplicationresourceareusedfortwodistinctpurposes.

Thefirstuseistheauthenticationofthesyncmastersatthedestinationwiththesyncmastersatthesource.Thisisalwaysdoneusingaclientauthenticationcertificatewhichisfoundinatls.keyfilefieldinasecretidentifiedbyspec.source.auth.keyfileSecretName.

TheseconduseistheauthenticationoftheArangoDBReplicationoperatorwiththesyncmastersatthesourceordestination.Theseconnectionsaremadetoconfiguresynchronization,stopconfigurationandfetchthestatusoftheconfiguration.Themethodusedforthisauthenticationisderivedasfollows(whereXiseithersourceordestination):

Ifspec.X.userSecretNameisset,theusername+passwordfoundintheSecretidentifiedbythisnameisused.Ifspec.X.keyfileSecretNameisset,theclientauthenticationcertificate(keyfile)foundintheSecretidentifierbythisnameisused.Ifspec.X.deploymentNameisset,theJWTsecretfoundinthedeploymentisused.

Creatingclientauthenticationcertificatekeyfiles

TheclientauthenticationcertificatesneededfortheSecretsidentifiedbyspec.source.auth.keyfileSecretName&spec.destination.auth.keyfileSecretNamearenormalArangoDBkeyfilesthatcanbecreatedbythearangosynccreateclient-authkeyfilecommand.Inordertodoso,youmusthaveaccesstotheclientauthenticationCAofthesource/destination.

IftheclientauthenticationCAatthesource/destinationalsocontainsaprivatekey(ca.key),theArangoDeploymentoperatorcanbeusedtocreatesuchakeyfileforyou,withouttheneedtohavearangosyncinstalledlocally.Readthefollowingparagraphsforinstructionsonhowtodothat.

CreatingandusingaccesspackagesAnaccesspackageisaYAMLfilethatcontains:

Aclientauthenticationcertificate,wrappedinaSecretinatls.keyfiledatafield.ATLScertificateauthoritypublickey,wrappedinaSecretinaca.crtdatafield.

TheformatoftheaccesspackageissuchthatitcanbeinsertedintoaKubernetesclusterusingthestandardkubectltool.

TocreateanaccesspackagethatcanbeusedtoauthenticatewiththeArangoDBSyncMastersofanArangoDeployment,addanameofanon-existingSecrettothespec.sync.externalAccess.accessPackageSecretNamesfieldoftheArangoDeployment.Inresponse,aSecretiscreatedinthatKubernetescluster,withthegivenname,thatcontainsaaccessPackage.yamldatafieldthatcontainsaKubernetesresourcespecificationthatcanbeinsertedintotheotherKubernetescluster.

Theprocessforcreatingandusinganaccesspackageforauthenticationatthesourceclusterisasfollows:

EdittheArangoDeploymentresourceofthesourcecluster,setspec.sync.externalAccess.accessPackageSecretNamesto["my-access-package"]

WaitfortheArangoDeploymentoperatortocreateaSecretnamedmy-access-package.ExtracttheaccesspackagefromtheKubernetessourceclusterusing:


355

kubectlgetsecretmy-access-package--template='{{index.data"accessPackage.yaml"}}'|base64-D>accessPackage.yaml

InsertthesecretsfoundintheaccesspackageintheKubernetesdestinationclusterusing:

kubectlapply-faccessPackage.yaml

Asaresult,thedestinationKubernetesclusterwillhave2additionalSecrets.Onecontainsaclientauthenticationcertificateformattedasakeyfile.AnothercontainsthepublickeyoftheTLSCAcertificateofthesourcecluster.


356

StorageAnArangoDBclusterreliesheavilyonfastpersistentstorage.TheArangoDBKubernetesOperatorusesPersistentVolumeClaimstodeliverthestoragetoPodsthatneedthem.

Storageconfiguration

IntheArangoDeploymentresource,onecanspecifythetypeofstorageusedbygroupsofserversusingthespec.<group>.storageClassNamesetting.

ThisisanexampleofaClusterdeploymentthatstoresitsagent&dbserverdataonPersistentVolumesthatusethemy-local-ssdStorageClass



metadata:

name:"cluster-using-local-ssh"

spec:

mode:Cluster

agents:

storageClassName:my-local-ssd

dbservers:

storageClassName:my-local-ssd

Theamountofstorageneededisconfiguredusingthespec.<group>.resources.requests.storagesetting.

Notethatconfiguringstorageisdonepergroupofservers.Itisnotpossibletoconfigurestorageperindividualserver.

ThisisanexampleofaClusterdeploymentthatrequestsvolumesof80GBforeverydbserver,resultinginatotalstoragecapacityof240GB(with3dbservers).



metadata:

name:"cluster-using-local-ssh"

spec:

mode:Cluster

dbservers:

resources:

requests:

storage:80Gi

Localstorage

Foroptimalperformance,ArangoDBshouldbeconfiguredwithlocallyattachedSSDstorage.

TheeasiestwaytoaccomplishthisistodeployanArangoLocalStorageresource.TheArangoDBStorageOperatorwilluseittoprovidePersistentVolumesforyou.

ThisisanexampleofanArangoLocalStorageresourcethatwillresultinPersistentVolumescreatedonanynodeoftheKubernetesclusterunderthedirectory/mnt/big-ssd-disk.

apiVersion:"storage.arangodb.com/v1alpha"

kind:"ArangoLocalStorage"

metadata:

name:"example-arangodb-storage"

spec:

storageClass:

name:my-local-ssd

localPath:

-/mnt/big-ssd-disk

Storage

357

NotethatusinglocalstoragerequiredVolumeSchedulingtobeenabledinyourKubernetescluster.ONKubernetes1.10thisisenabledbydefault,onversion1.9youhavetoenableitwitha--feature-gatesetting.

ManuallycreatingPersistentVolumes

ThealternativeistocreatePersistentVolumesmanually,forallserversthatneedpersistentstorage(single,agents&dbservers).E.g.foraClusterwith3agentsand5dbservers,youmustcreate8volumes.

Notethateachvolumemusthaveacapacitythatisequaltoorhigherthanthecapacityneededforeachserver.

Toselectthecorrectnode,addarequirednode-affinityannotationasshownintheexamplebelow.

apiVersion:v1

kind:PersistentVolume

metadata:

name:volume-agent-1

annotations:

"volume.alpha.kubernetes.io/node-affinity":'{

"requiredDuringSchedulingIgnoredDuringExecution":{

"nodeSelectorTerms":[

{"matchExpressions":[

{"key":"kubernetes.io/hostname",

"operator":"In",

"values":["node-1"]

}

]}

]}

}'

spec:

capacity:

storage:100Gi

accessModes:

-ReadWriteOnce

persistentVolumeReclaimPolicy:Delete

storageClassName:local-ssd

local:

path:/mnt/disks/ssd1

ForKubernetes1.9andup,youshouldcreateaStorageClasswhichisconfiguredtobindvolumesontheirfirstuseasshownintheexamplebelow.ThisensuresthattheKubernetesschedulertakesallconstraintsonaPodthatintoconsiderationbeforebindingthevolumetoaclaim.

kind:StorageClass

apiVersion:storage.k8s.io/v1

metadata:

name:local-ssd

provisioner:kubernetes.io/no-provisioner

volumeBindingMode:WaitForFirstConsumer

Storage

358

ArangoLocalStorageCustomResourceTheArangoDBStorageOperatorcreatesandmaintainsArangoDBstorageresourcesinaKubernetescluster,givenastoragespecification.ThisstoragespecificationisaCustomResourcefollowingaCustomResourceDefinitioncreatedbytheoperator.

Exampleminimalstoragedefinition:

apiVersion:"storage.arangodb.com/v1alpha"

kind:"ArangoLocalStorage"

metadata:

name:"example-arangodb-storage"

spec:

storageClass:

name:my-local-ssd

localPath:

-/mnt/big-ssd-disk


aStorageClasscalledmy-local-ssdthedynamicprovisioningofPersistentVolume'swithalocalvolumeonanodewherethelocalvolumestartsinasub-directoryof/mnt/big-ssd-disk.thedynamiccleanupofPersistentVolume's(createdbytheoperator)afteroneisreleased.

Theprovisionedvolumeswillhaveacapacitythatmatchestherequestedcapacityofvolumeclaims.

SpecificationreferenceBelowyou'llfindallsettingsoftheArangoLocalStoragecustomresource.

spec.storageClass.name:string

ThissettingspecifiesthenameofthestorageclassthatcreatedPersistentVolumewilluse.

Ifempty,thisfielddefaultstothenameoftheArangoLocalStorageobject.

IfaStorageClasswithgivennamedoesnotyetexist,itwillbecreated.

spec.storageClass.isDefault:bool

ThissettingspecifiesifthecreatedStorageClasswillbemarkedasdefaultstorageclass.(defaultisfalse)

spec.localPath:stringList

Thissettingspecifiesoneofmorelocaldirectories(onthenodes)usedtocreatepersistentvolumesin.

spec.nodeSelector:nodeSelector

Thissettingspecifieswhichnodestheoperatorwillprovisionpersistentvolumeson.

StorageResource

359

Secureconnections(TLS)TheArangoDBKubernetesOperatorwillbydefaultcreateArangoDBdeploymentsthatusesecureTLSconnections.

ItusesasingleCAcertificate(storedinaKubernetessecret)andonecertificateperArangoDBserver(storedinaKubernetessecretperserver).

TodisableTLS,setspec.tls.caSecretNametoNone.

InstallCAcertificateIftheCAcertificateisself-signed,itwillnotbetrustedbybrowsers,untilyouinstallitinthelocaloperatingsystemorbrowser.Thisprocessdiffersperoperatingsystem.

Todoso,youfirsthavetofetchtheCAcertificatefromitsKubernetessecret.

kubectlgetsecret<deploy-name>-ca--template='{{index.data"ca.crt"}}'|base64-D>ca.crt

Windows

ToinstallaCAcertificateinWindows,followtheproceduredescribedhere.

MacOS

ToinstallaCAcertificateinMacOS,run:

sudo/usr/bin/securityadd-trusted-cert-d-rtrustRoot-k/Library/Keychains/System.keychainca.crt

TouninstallaCAcertificateinMacOS,run:

sudo/usr/bin/securityremove-trusted-cert-dca.crt

Linux

ToinstallaCAcertificateinLinux,onUbuntu,run:

sudocpca.crt/usr/local/share/ca-certificates/<some-name>.crt

sudoupdate-ca-certificates

Seealso

Authentication

TLS

360

http://wiki.cacert.org/HowTo/InstallCAcertRoots

TroubleshootingWhileKubernetesandtheArangoDBKubernetesoperatorwillautomaticallyresolvealotofissues,therearealwayscaseswherehumanattentionisneeded.

Thischaptergivesyourtips&trickstohelpyoutroubleshootdeployments.

Wheretolook

InKubernetesallresourcescanbeinspectedusingkubectlusingeitherthegetordescribecommand.

Togetalldetailsoftheresource(bothspecification&status),runthefollowingcommand:

kubectlget<resource-type><resource-name>-n<namespace>-oyaml

Forexample,togettheentirespecificationandstatusofanArangoDeploymentresourcenamedmy-arangodbinthedefaultnamespace,run:

kubectlgetArangoDeploymentmy-arango-ndefault-oyaml

#orshorter

kubectlgetarangomy-arango-oyaml

Severaltypesofresources(includingallArangoDBcustomresources)supportevents.Theseeventsshowwhathappenedtotheresourceovertime.

Toshowtheevents(andmostimportantresourcedata)ofaresource,runthefollowingcommand:

kubectldescribe<resource-type><resource-name>-n<namespace>

GettinglogsAnotherinvaluablesourceofinformationisthelogofcontainersbeingruninKubernetes.TheselogsareaccessiblethroughthePodsthatgroupthesecontainers.

TofetchthelogsofthedefaultcontainerrunninginaPod,run:

kubectllogs<pod-name>-n<namespace>

#orwithfollowoptiontokeepinspectinglogswhiletheyarewritten

kubectllogs<pod-name>-n<namespace>-f

ToinspectthelogsofaspecificcontainerinPod,add-c<container-name>.YoucanfindthenamesofthecontainersinthePod,usingkubectldescribepod....

NotethattheArangoDBoperatorsarebeingdeployedthemselvesasaKubernetesDeploymentwith2replicas.Thismeansthatyouwillhavetofetchthelogsof2Podsrunningthosereplicas.

Whatif

ThePodsofadeploymentstayinPendingstate

Therearetwocommoncausesforthis.

Troubleshooting

361

1)ThePodscannotbescheduledbecausetherearenotenoughnodesavailable.Thisisusallyonlythecasewithaspec.environmentsettingthathasavalueofProduction.

Solution:Addmorenodes.

1)TherearenoPersistentVolumesavailabletobeboundtothePersistentVolumeClaimscreatedbytheoperator.

Solution:

UsekubectlgetpersistentvolumestoinspecttheavailablePersistentVolumesandifneeded,usetheArangoLocalStorageoperatortoprovisionPersistentVolumes.

WhenrestartingaNode,thePodsscheduledonthatnoderemaininTerminatingstate

WhenaNodenolongermakesregularcallstotheKubernetesAPIserver,itismarkedasnotavailable.DependingonspecificsettingsinyourPods,KuberneteswillatsomepointdecidetoterminatethePod.AslongastheNodeisnotcompletelyremovedfromtheKubernetesAPIserver,KuberneteswilltrytousetheNodeitselftoterminatethePod.

TheArangoDeploymentoperatorrecognizesthisconditionandwilltrytoreplacethosePodswithPodsondifferentnodes.Theexactbehaviordifferspertypeofserver.

WhathappenswhenaNodewithlocaldataisbroken

WhenaNodewithPersistentVolumeshostedonthatNodeisbrokenandcannotberepaired,thedatainthosePersistentVolumesislost.

IfanArangoDeploymentoftypeSinglewasusingoneofthosePersistentVolumesthedatabaseislostandmustberestoredfromabackup.

IfanArangoDeploymentoftypeActiveFailoverorClusterwasusingoneofthosePersistentVolumes,itdependsonthetypeofserverthatwasusingthevolume.

IfanAgentwasusingthevolume,itcanberepairedaslongas2otheragentsarestillhealthy.IfaDBServerwasusingthevolume,andthereplicationfactorofalldatabasecollectionsis2orhigher,andtheremainingdbserversarestillhealthy,theclusterwillduplicatetheremainingreplicastobringthenumberofreplicasesbacktotheoriginalnumber.IfaDBServerwasusingthevolume,andthereplicationfactorofadatabasecollectionis1andhappenstobestoredonthatdbserver,thedataislost.IfasingleserverofanActiveFailoverdeploymentwasusingthevolume,andtheothersingleserverisstillhealthy,theothersingleserverwillbecomeleader.Afterreplacingthefailedsingleserver,thenewfollowerwillsynchronizewiththeleader.

Troubleshooting

362

AdministrationMostadministrationcanbemanagedusingthearangosh.

FilesystemsAsonewouldexpectforadatabase,werecommendalocallymountedfilesystems.

NFSorsimilarnetworkfilesystemswillnotwork.

OnLinuxwerecommendtheuseofext4fs,onWindowsNTFSandonMacOSHFS+.

WerecommendtonotuseBTRFSonLinux.ItisknowntonotworkwellinconjunctionwithArangoDB.WeexperiencedthatArangoDBfaceslatencyissuesonaccessingitsdatabasefilesonBTRFSpartitions.InconjunctionwithBTRFSandAUFSwealsosawdatalossonrestart.

Administration

363

WebInterfaceArangoDBcomeswithabuilt-inwebinterfaceforadministration.Theinterfacediffersforstandaloneinstancesandclustersetups.

Standalone:

Cluster:

WebInterface

364

WebInterface

365

DashboardTheDashboardtabprovidesstatisticswhicharepolledregularlyfromtheArangoDBserver.

RequestsStatistics:

RequestspersecondRequesttypesNumberofclientconnectionsTransfersizeTransfersize(distribution)AveragerequesttimeAveragerequesttime(distribution)

SystemResources:

NumberofthreadsMemoryVirtualsizeMajorpagefaultsUsedCPUtime

Replication:

ReplicationstateTotalsTicksProgress

Dashboard

366

ClusterTheclustersectiondisplaysstatisticsaboutthegeneralclusterperformance.

Statistics:

AvailableandmissingcoordinatorsAvailableandmissingdatabaseserversMemoryusage(percent)CurrentconnectionsData(bytes)HTTP(bytes)Averagerequesttime(seconds)

Nodes

Overview

Theoverviewshowsavailableandmissingcoordinatorsanddatabaseservers.

Cluster

367

Functions:

CoordinatorDashboard:ClickonaCoordinatorwillopenastatisticsdashboard.

Information(Coordinator/Databaseservers):

NameEndpointLastHeartbeatStatusHealth

Shards

Theshardsectiondisplaysallavailableshardedcollections.

Cluster

368

Functions:

MoveShardLeader:Clickonaleaderdatabaseofashardserverwillopenamovesharddialog.Shardscanbetransferredtoallavailabledatabasservers,excepttheleadingdatabaseserveroranavailablefollower.MoveShardFollower:Clickonafollowerdatabaseofashardwillopenamovesharddialog.Shardscanbetransferredtoallavailabledatabasservers,excepttheleadingdatabaseserveroranavailablefollower.RebalanceShards:Anewdatabaseserverwillnothaveanyshards.Withtherebalancefunctionalitytheclusterwillstarttorebalanceshardsincludingemptydatabaseservers.

Information(collection):

ShardLeader(greenstate:synciscomplete)Followers

Cluster

369

CollectionsThecollectionssectiondisplaysallavailablecollections.Fromhereyoucancreatenewcollectionsandjumpintoacollectionfordetails(clickonacollectiontile).

Functions:

A:TogglefilterpropertiesB:SearchcollectionbynameD:CreatecollectionC:FilterpropertiesH:Showcollectiondetails(clicktile)

Information:

E:CollectiontypeF:Collectionstate(unloaded,loaded,...)G:Collectionname

Collection

Collections

370

Therearefourviewcategories:

1. Content:

CreateadocumentDeleteadocumentFilterdocumentsDownloaddocumentsUploaddocuments

2. Indices:

CreateindicesDeleteindices

3. Info:

Detailedcollectioninformationandstatistics4. Settings:

Configurename,journalsize,indexbuckets,waitforsyncDeletecollectionTruncatecollectionUnload/LoadcollectionSavemodifedproperties(name,journalsize,indexbuckets,waitforsync)

Additionalinformation:

Uploadformat:

I.Line-wise

{"_key":"key1",...}

{"_key":"key2",...}

II.JSONdocumentsinalist

[

{"_key":"key1",...},

{"_key":"key2",...}

Collections

371

]

Collections

372

DocumentThedocumentsectionoffersaeditorwhichletyoueditdocumentsandedgesofacollection.

Functions:

EditdocumentSavedocumentDeletedocmentSwitchbetweenTree/Code-ModeCreateanewdocument

Information:

Displays:_id,_rev,_keyproperties

Document

373

QueryViewThequeryviewoffersyouthreedifferentsubviews:

EditorRunningQueriesSlowQueryHistory

AQLQueryEditor

ThewebinterfaceoffersaAQLQueryEditor:

Theeditorissplitintotwoparts,thequeryeditorpaneandthebindparameterpane.

Theleftpaneisyourregularqueryinputfield,whereyoucaneditandthenexecuteorexplainyourqueries.Bydefault,theenteredbindparameterwillautomaticallyberecognizedandshowninthebindparametertableintherightpane,whereyoucaneasilyeditthem.

Theinputfieldsareequippedwithtypedetection.Thismeansyoudon'thavetousequotemarksaroundstring,justwritethemas-is.Numberswillbetreatedasnumbers,trueandfalseasbooleans,nullasnull-typevalue.Squarebracketscanbeusedtodefinearrays,andcurlybracesforobjects(keysandvalueshavetobesurroundedbydoublequotes).Thiswillmostlybewhatyouwant.Butifyouwanttoforcesomethingtobetreatedasstring,usequotationmarksforthevalue:

123//interpretedasnumber

"123"//interpretedasstring

["foo","bar",123,true]//interpretedasarray

['foo','bar',123,true]//interpretedasstring

Queries

374

IfyouareusedtoworkwithJSON,youmaywanttoswitchthebindparametereditortoJSONmodebyclickingontheupperrighttogglebutton.YoucantheneditthebindparametersinrawJSONformat.

CustomQueries

TosavethecurrentqueryusetheSavebuttoninthetopleftcorneroftheeditororusetheshortcut(seebelow).

BypressingtheQueriesbuttoninthetopleftcorneroftheeditoryouactivatethecustomqueriesview.Hereyoucanselectapreviouslystoredcustomqueryoroneofourqueryexamples.

Clickonaquerytitletogetacodepreview.Inaddition,thereareactionbuttonsto:

CopytoeditorExplainqueryRunqueryDeletequery

Forthebuilt-inexamplequeries,thereisonlyCopytoeditoravailable.

ToexportorimportqueriestoandfromJSONyoucanusethebuttonsontheright-handside.

Result

Queries

375

Eachqueryyouexecuteorexplainopensupanewresultbox,soyouareabletofireupmultiplequeriesandviewtheirresultsatthesametime.Everyqueryresultboxgivesyoudetailedqueryinformationandofcoursethequeryresultitself.Theresultboxescanbedismissedindividually,oraltogetherusingtheRemoveresultsbutton.ThetogglebuttoninthetoprightcornerofeachboxswitchesbackandforthbetweentheResultandAQLquerywithbindparameters.

Spotlight

Thespotlightfeatureopensupamodalview.ThereyoucanfindallAQLkeywords,AQLfunctionsandcollections(filteredbytheirtype)tohelpyoutobemoreproductiveinwritingyourqueries.Spotlightcanbeopenedbythemagicwandiconinthetoolbarorviashortcut(seebelow).

AQLEditorShortcuts

Queries

376

Ctrl/Cmd+ReturntoexecuteaqueryCtrl/Cmd+Shift+ReturntoexplainaqueryCtrl/Cmd+Shift+StosavethecurrentqueryCtrl/Cmd+Shift+CtotogglecommentsCtrl+SpacetoopenupthespotlightsearchCtrl+Cmd+ZtoundolastchangeCtrl+Cmd+Shift+Ztoredolastchange

RunningQueries

TheRunningQueriestabgivesyouacompactoverviewofallrunningqueries.Byclickingtheredminusbutton,youcanaborttheexecutionofarunningquery.

SlowQueryHistory

Queries

377

TheSlowQueryHistorytabgivesyouacompactoverviewofallpastslowqueries.

Queries

378

GraphsTheGraphstabprovidesaviewerfacilityforgraphdatastoredinArangoDB.ItallowsbrowsingArangoDBgraphsstoredinthe_graphssystemcollectionoragraphconsistingofanarbitraryvertexandedgecollection.

Pleasenotethatthegraphviewerrequirescanvas(optional:webgl)supportinyourbrowser.EspeciallyInternetExplorerbrowsersolderthanversion9arelikelytonotsupportthis.

GraphViewer

Graphs

379

TopToolbarFunctions:

Loadfullgraph(Alsonodeswithoutconnectionswillbedrawn.Usefulduringgraphmodelingsetup)TakeagraphscreenshotStartfullscreenmodeOpengraphoptionsmenu

DefaultContextMenu(mouse-clickbackground):

AddanewnodeClosevisiblecontextmenu(s)

NodeContextMenu(mouse-clicknode):

DeletenodeEditnodeExpandnode(Showallboundedges)Drawedge(Connectwithanothernode)Setasstartnode(TheGraphwillrerenderstartingtheselectednodeandgivenoptions(graphoptionsmenu))

EdgeContextMenu(mouse-clickedge):

EditedgeDeleteedge

EdgeHighlighting(right-mouse-clicknode):

Highlightalledgesconnectedtothenode(right-clickatthebackgroundwillremovehighlighting)

Graphs

380

GraphViewerOptionsGraphOptionsMenu:

Startnode(string-validnodeidorspaceseperatedlistofid's):Heartofyourgraph.Renderingandtraversingwillstartfromhere.Emptyvaluemeans:arandomstartingpointwillbeused.Layout:Differentgraphlayoutingalgoritms.Nooverlap(optimal:biggraph),forcelayout(optimal:mediumgraph),fruchtermann(optimal:littletomediumgraph).Renderer:Canvasmodeallowsediting.WebGLcurrentlyoffersonlydisplaymode(alotfasterwithmuchnodes/edges).Searchdepth(number):Searchdepthwhichisstartingfromyourstartnode.Limit(number):Limitnodescount.Ifemptyorzero,nolimitisset.

NodesOptionsMenu:

Label(string):Nodeswillbelabeledbythisattribute.Ifnodeattributeisnotfound,nolabelwillbedisplayed.AddCollectionName:Thisappendsthecollectionnametothelabel,ifitexists.ColorByCollections:Shouldnodesbecolorizedbytheircollection?Ifenabled,nodecolorandnodecolorattributewillbeignored.Color:Defaultnodecolor.ColorAttribute(string):Ifanattributeisgiven,nodeswillthenbecolorizedbytheattribute.Thissettingignoresdefaultnodecolorifset.SizeByConnections:Shouldnodesbesizedbytheiredgescount?Ifenabled,nodesizingattributewillbeignored.SizingAttribute(number):Defaultnodesize.Numericvalue>0.

EdgesOptionsMenu:

Label(string):Edgeswillbelabeledbythisattribute.Ifedgeattributeisnotfound,nolabelwillbedisplayed.AddCollectionName:Thisappendsthecollectionnametothelabel,ifitexists.ColorByCollections:Shouldedgesbecolorizedbytheircollection?Ifenabled,edgecolorandedgecolorattributewillbeignored.Color:Defaultedgecolor.ColorAttribute(string):Ifanattributeisgiven,edgeswillthenbecolorizedbytheattribute.Thissettingignoresdefaultnodecolorifset.Type:Therendereroffersmultipletypesofrendering.Theyonlydifferintheirdisplaystyle,exceptforthetype'curved'.Thecurvedtypeallowstodisplaymorethanoneedgesbetweentwonodes.

Graphs

381

Graphs

382

ServicesTheservicessectiondisplaysallinstalledfoxxapplications.Youcancreatenewservicesorgointoadetailedviewofachoosenservice.

CreateService

Therearefourdifferentpossibilitiestocreateanewservice:

1. Createserviceviazipfile2. Createserviceviagithubrepository3. CreateserviceviaofficialArangoDBstore4. Createablankservicefromscratch

Services

383

ServiceView

Thissectionoffersseveralinformationaboutaspecificservice.

Therearefourviewcategories:

1. Info:

Displaysname,shortdescription,license,version,mode(production,development)Offersabuttontogototheservicesinterface(ifavailable)

Services

384

2. Api:

DisplayAPIasSwaggerUIDisplayAPIasRAWJSON

3. Readme:

Displaystheservicesmanual(ifavailable)4. Settings:

DownloadserviceaszipfileRunservicetests(ifavailable)Runservicescripts(ifavailable)Configuredependencies(ifavailable)Changeserviceparameters(ifavailable)Changemode(production,development)ReplacetheserviceDeletetheservice

Services

385

ManagingUsersintheWebInterfaceArangoDBusersaregloballystoredinthe_systemdatabaseandcanonlybemanangedwhileloggedontothisdatabase.ThereyoucanfindtheUserssection:

General

SelectausertobringuptheGeneraltabwiththeusername,nameandactivestatus,aswellasoptionstodeletetheuserorchangethepassword.

Users

386

Permissions

SelectauserandgotothePermissionstab.Youwillseealistofdatabasesandtheircorrespondingdatabaseaccesslevelforthatuser.

Users

387

Pleasenotethatserveraccesslevelfollowsfromtheaccesslevelonthedatabase_system.Furthermore,thedefaultdatabaseaccesslevelforthisuserappearintheartificialrowwiththedatabasename*.

Belowthistableisanotheroneforthecollectioncategoryaccesslevels.Atfirst,itshowsthelistofdatabases,too.Ifyouclickonadatabase,thelistofcollectionsinthatdatabasewillbeopenandyoucanseethedefinedcollectionaccesslevelsforeachcollectionofthatdatabase(whichcanbeallunselectedwhichmeansthatnothingisexplicitlyset).Thedefaultaccesslevelsforthisuseranddatabaseappearintheartificialrowwiththecollectionname*.

AlsoseeManagingUsersaboutaccesslevels.

Users

388

LogsThelogssectiondisplaysallavailablelogentries.Logentriesarefilterablebytheirlogleveltypes.

Functions:

Filterlogentriesbyloglevel(all,info,error,warning,debug)

Information:

LoglevelDateMessage

Logs

389

ArangoDBShellIntroductionTheArangoDBshell(arangosh)isacommand-linetoolthatcanbeusedforadministrationofArangoDB,includingrunningad-hocqueries.

ThearangoshbinaryisshippedwithArangoDB.ItoffersaJavaScriptshellenvironmentprovidingaccesstotheArangoDBserver.Arangoshcanbeinvokedlikethis:

unix>arangosh

BydefaultarangoshwilltrytoconnecttoanArangoDBserverrunningonserverlocalhostonport8529.Itwillusetheusernamerootandanemptypasswordbydefault.Additionallyitwillconnecttothedefaultdatabase(_system).Allthesedefaultscanbechangedusingthefollowingcommand-lineoptions:

--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:databaseusername--server.password:passwordtousewhenconnecting--server.authentication:whetherornottouseauthentication

Forexample,toconnecttoanArangoDBserveronIP192.168.173.13onport8530withtheuserfooandusingthedatabasetest,use:

unix>arangosh\

--server.endpointtcp://192.168.173.13:8530\

--server.usernamefoo\

--server.databasetest\

--server.authenticationtrue

arangoshwillthendisplayapasswordpromptandtrytoconnecttotheserverafterthepasswordwasentered.

Tochangethecurrentdatabaseaftertheconnectionhasbeenmade,youcanusethedb._useDatabase()commandinarangosh:

arangosh>db._createDatabase("myapp");

true

arangosh>db._useDatabase("myapp");

true

arangosh>db._useDatabase("_system");

true

arangosh>db._dropDatabase("myapp");

true

Togetalistofavailablecommands,arangoshprovidesahelp()function.Callingitwilldisplayhelpfulinformation.

arangoshalsoprovidesauto-completion.Additionalinformationonavailablecommandsandmethodsisthusprovidedbytypingthefirstfewlettersofavariableandthenpressingthetabkey.Itisrecommendtotrythiswithenteringdb.(withoutpressingreturn)andthenpressingtab.

Bytheway,arangoshprovidesthedbobjectbydefault,andthisobjectcanbeusedforswitchingtoadifferentdatabaseandmanagingcollectionsinsidethecurrentdatabase.

Foralistofavailablemethodsforthedbobject,type

arangosh>db._help();

showexecutionresultsyoucanpastemultiplelinesintoarangosh,giventhefirstlineendswithanopeningbrace:

arangosh>for(vari=0;i<10;i++){

ArangoDBShell

390

........>require("@arangodb").print("Helloworld"+i+"!\n");

........>}

showexecutionresultsToloadyourownJavaScriptcodeintothecurrentJavaScriptinterpretercontext,usetheloadcommand:

require("internal").load("/tmp/test.js")//<-Linux/MacOS

require("internal").load("c:\\tmp\\test.js")//<-Windows

Exitingarangoshcanbedoneusingthekeycombination<CTRL>+Dorbytypingquit<CR>

Escaping

InAQL,escapingisdonetraditionallywiththebackslashcharacter:\.Asseenabove,thisleadstodoublebackslasheswhenspecifyingWindowspaths.Arangoshrequiresanotherlevelofescaping,alsowiththebackslashcharacter.ItaddsuptofourbackslashesthatneedtobewritteninArangoshforasingleliteralbackslash(c:\tmp\test.js):

db._query('RETURN"c:\\\\tmp\\\\test.js"')

Youcanusebindvariablestomitigatethis:

varsomepath="c:\\tmp\\test.js"

db._query(aql`RETURN${somepath}`)

ArangoDBShell

391

ArangoDBShellOutputTheArangoDBshellwillprinttheoutputofthelastevaluatedexpressionbydefault:

arangosh>42*23

966

Inordertopreventprintingtheresultofthelastevaluatedexpression,theexpressionresultcanbecapturedinavariable,e.g.

arangosh>varcalculationResult=42*23

ThereisalsotheprintfunctiontoexplicitlyprintoutvaluesintheArangoDBshell:

arangosh>print({a:"123",b:[1,2,3],c:"test"});

showexecutionresultsBydefault,theArangoDBshellusesaprettyprinterwhenJSONdocumentsareprinted.Thisensuresdocumentsareprintedinahuman-readableway:

arangosh>db._create("five")

arangosh>for(i=0;i<5;i++)db.five.save({value:i})

arangosh>db.five.toArray()

showexecutionresultsWhilethepretty-printerproducesnicelookingresults,itwillneedalotofscreenspaceforeachdocument.Sometimesamoredenseoutputmightbebetter.Inthiscase,theprettyprintercanbeturnedoffusingthecommandstop_pretty_print().

Toturnonprettyprintingagain,usethestart_pretty_print()command.

ShellOutput

392

ArangoDBShellConfigurationarangoshwilllookforauser-definedstartupscriptnamed.arangosh.rcintheuser'shomedirectoryonstartup.Thehomedirectorywilllikelybe/home/<username>/onUnix/Linux,andisdeterminedonWindowsbypeekingintotheenvironmentvariables%HOMEDRIVE%and%HOMEPATH%.

Ifthefile.arangosh.rcispresentinthehomedirectory,arangoshwillexecutethecontentsofthisfileinsidetheglobalscope.

Youcanusethistodefineyourownextravariablesandfunctionsthatyouneedoften.Forexample,youcouldputthefollowingintothe.arangosh.rcfileinyourhomedirectory:

//"var"keywordavoidedintentionally...

//otherwise"timed"wouldnotsurvivethescopeofthisscript

global.timed=function(cb){

console.time("callback");

cb();

console.timeEnd("callback");

};

Thiswillmakeafunctionnamedtimedavailableinarangoshintheglobalscope.

Youcannowstartarangoshandinvokethefunctionlikethis:

timed(function(){

for(vari=0;i<1000;++i){

db.test.save({value:i});

}

});

Pleasekeepinmindthat,ifpresent,the.arangosh.rcfileneedstocontainvalidJavaScriptcode.Ifyouwantanyvariablesintheglobalscopetosurviveyouneedtoomitthevarkeywordforthem.Otherwisethevariableswillonlybevisibleinsidethescriptitself,butnotoutside.

Configuration

393

DetailsabouttheArangoDBShellAftertheserverhasbeenstarted,youcanusetheArangoDBshell(arangosh)toadministratetheserver.Withoutanyarguments,theArangoDBshellwilltrytocontacttheserveronport8529onthelocalhost.FormoreinformationseetheArangoDBShelldocumentation.Youmightneedtosetadditionaloptions(endpoint,usernameandpassword)whenconnecting:

unix>./arangosh--server.endpointtcp://127.0.0.1:8529--server.usernameroot

TheshellwillprintitsownversionnumberandifsuccessfullyconnectedtoaservertheversionnumberoftheArangoDBserver.

Command-LineOptionsUse--helptogetalistofcommand-lineoptions:

unix>./arangosh--help

STANDARDoptions:

--audit-log<string>auditlogfiletosavecommandsandresultsto

--configuration<string>readconfigurationfile

--helphelpmessage

--max-upload-size<uint64>maximumsizeofimportchunks(inbytes)(default:500000)

--no-auto-completedisableautocompletion

--no-colorsdeactivatecolorsupport

--pager<string>outputpager(default:"less-X-R-F-L")

--pretty-printprettyprintvalues

--quietnobanner

--temp.path<string>pathfortemporaryfiles(default:"/tmp/arangodb")

--use-pagerusepager

JAVASCRIPToptions:

--javascript.check<string>syntaxcheckcodeJavaScriptcodefromfile

--javascript.execute<string>executeJavaScriptcodefromfile

--javascript.execute-string<string>executeJavaScriptcodefromstring

--javascript.startup-directory<string>startuppathscontainingtheJavaScriptfiles

--javascript.unit-tests<string>donotstartasshell,rununittestsinstead

--jslint<string>donotstartasshell,runjslintinstead

LOGGINGoptions:

--log.level<string>loglevel(default:"info")

CLIENToptions:

--server.connect-timeout<double>connecttimeoutinseconds(default:3)

--server.authentication<bool>whetherornottouseauthentication(default:true)

--server.endpoint<string>endpointtoconnectto,use'none'tostartwithoutaserver(default:"tcp://127.0

.0.1:8529")

--server.password<string>passwordtousewhenconnecting(leaveemptyforprompt)

--server.request-timeout<double>requesttimeoutinseconds(default:300)

--server.username<string>usernametousewhenconnecting(default:"root")

DatabaseWrappersThedbobjectisavailableinarangoshaswellasonarangodi.e.ifyou'reusingFoxx.Whileitsinterfaceispersistantbetweenthearangoshandthearangodimplementations,itsunderpinningisnot.ThearangodimplementationareJavaScriptwrappersaroundArangoDB'snativeC++implementation,whereasthearangoshimplementationwrapsHTTPaccessestoArangoDB'sRESTfullAPI.

Sowhilethiscodemayproducesimilarresultswhenexecutedinarangoshandarangod,thecpuusageandtimerequiredwillbereallydifferent:

for(i=0;i<100000;i++){

db.test.save({name:{first:"Jan"},count:i});

}

Sincethearangoshversionwillbedoingaround100kHTTPrequests,andthearangodversionwilldirectlywritetothedatabase.

Details

394

Using arangoshviaunixshebangmechanisms

Inunixoperatingsystemsyoucanstartscriptsbyspecifyingtheinterpreterinthefirstlineofthescript.Thisiscommonlycalledshebangorhashbang.Youcanalsodothatwitharangosh,i.e.create~/test.js:

#!/usr/bin/arangosh--javascript.execute

require("internal").print("helloworld")

db._query("FORxINtestRETURNx").toArray()

Notethatthefirstlinehastoendwithablankinordertomakeitwork.MarkitexecutabletotheOS:

#>chmoda+x~/test.js

andfinalytryitout:

#>~/test.js

Details

395

ArangoimpThismanualdescribestheArangoDBimporterarangoimp,whichcanbeusedforbulkimports.

ThemostconvenientmethodtoimportalotofdataintoArangoDBistousethearangoimpcommand-linetool.Itallowsyoutoimportdatarecordsfromafileintoanexistingdatabasecollection.

Itispossibletoimportdocumentkeyswiththedocumentsusingthe_keyattribute.Whenimportingintoanedgecollection,itismandatorythatallimporteddocumentshavethe_fromand_toattributes,andthattheycontainvalidreferences.

Let'sassumeforthefollowingexamplesyouwanttoimportuserdataintoanexistingcollectionnamed"users"ontheserver.

ImportingDataintoanArangoDBDatabase

ImportingJSON-encodedData

Let'sfurtherassumetheimportathandisencodedinJSON.We'llbeusingtheseexampleuserrecordstoimport:

{"name":{"first":"John","last":"Connor"},"active":true,"age":25,"likes":["swimming"]}

{"name":{"first":"Jim","last":"O'Brady"},"age":19,"likes":["hiking","singing"]}

{"name":{"first":"Lisa","last":"Jones"},"dob":"1981-04-09","likes":["running"]}

Toimporttheserecords,allyouneedtodoistoputthemintoafile(withonelineforeachrecordtoimport)andrunthefollowingcommand:

>arangoimp--file"data.json"--typejsonl--collection"users"

Thiswilltransferthedatatotheserver,importtherecords,andprintastatussummary.Toshowtheintermediateprogressduringtheimportprocess,theoption--progresscanbeadded.Thisoptionwillshowthepercentageoftheinputfilethathasbeensenttotheserver.Thiswillonlybeusefulforbigimportfiles.

>arangoimp--file"data.json"--typejson--collectionusers--progresstrue

Itisalsopossibletousetheoutputofanothercommandasaninputforarangoimp.Forexample,thefollowingshellcommandcanbeusedtopipedatafromthecatprocesstoarangoimp:

>catdata.json|arangoimp--file---typejson--collectionusers

Notethatyouhavetouse--file-ifyouwanttouseanothercommandasinputforarangoimp.Noprogresscanbereportedforsuchimportsasthesizeoftheinputwillbeunknowntoarangoimp.

Bydefault,theendpointtcp://127.0.0.1:8529willbeused.Ifyouwanttospecifyadifferentendpoint,youcanusethe--server.endpointoption.Youprobablywanttospecifyadatabaseuserandpasswordaswell.Youcandosobyusingtheoptions--server.usernameand--server.password.Ifyoudonotspecifyapassword,youwillbepromptedforone.

>arangoimp--server.endpointtcp://127.0.0.1:8529--server.usernameroot--file"data.json"--typejson--collection"users"

Notethatthecollection(usersinthiscase)mustalreadyexistortheimportwillfail.Ifyouwanttocreateanewcollectionwiththeimportdata,youneedtospecifythe--create-collectionoption.Notethatbydefaultitwillcreateadocumentcollectionandnoedecollection.

>arangoimp--file"data.json"--typejson--collection"users"--create-collectiontrue

Tocreateanedgecollectioninstead,usethe--create-collection-typeoptionandsetittoedge:

>arangoimp--file"data.json"--collection"myedges"--create-collectiontrue--create-collection-typeedge

Arangoimp

396

Whenimportingdataintoanexistingcollectionitisoftenconvenienttofirstremovealldatafromthecollectionandthenstarttheimport.Thiscanbeachievedbypassingthe--overwriteparametertoarangoimp.Ifitissettotrue,anyexistingdatainthecollectionwillberemovedpriortotheimport.Notethatanyexistingindexdefinitionsforthecollectionwillbepreservedevenif--overwriteissettotrue.

>arangoimp--file"data.json"--typejson--collection"users"--overwritetrue

AstheimportfilealreadycontainsthedatainJSONformat,attributenamesanddatatypesarefullypreserved.Ascanbeseenintheexampledata,thereisnoneedforalldatarecordstohavethesameattributenamesortypes.Recordscanbeinhomogeneous.

Pleasenotethatbydefault,arangoimpwillimportdataintothespecifiedcollectioninthedefaultdatabase(_system).Tospecifyadifferentdatabase,usethe--server.databaseoptionwheninvokingarangoimp.

Thetoolalsosupportsparallelimports,withmultiplethreads.Usingmultiplethreadsmayprovideaspeedup,especiallywhenusingtheRocksDBstorageengine.Tospecifythenumberofparallelthreadsusethe--threadsoption:

>arangoimp--threads4--file"data.json"--typejson--collection"users"

Notethatusingmultiplethreadsmayleadtoanon-sequentialimportoftheinputdata.Datathatappearslaterintheinputfilemaybeimportedearlierthandatathatappearsearlierintheinputfile.Thisisnormallynotaproblembutmaycauseissueswhenwhentherearedatadependenciesorduplicatesintheimportdata.Inthiscase,thenumberofthreadsshouldbesetto1.

JSONinputfileformats

Note:arangoimpsupportstwoformatswhenimportingJSONdatafromafile.Thefirstformatthatwealsousedaboveiscommonlyknownasjsonl).However,incontrasttotheJSONLspecificationitrequirestheinputfiletocontainonecompleteJSONdocumentineachline,e.g.

{"_key":"one","value":1}

{"_key":"two","value":2}

{"_key":"foo","value":"bar"}

...

SoonecouldarguethatthisisonlyasubsetofJSONL.

Theaboveformatcanbeimportedsequentiallybyarangoimp.Itwillreaddatafromtheinputfileinchunksandsenditinbatchestotheserver.Eachbatchwillbeaboutasbigasspecifiedinthecommand-lineparameter--batch-size.

AnalternativeistoputonebigJSONdocumentintotheinputfilelikethis:

[

{"_key":"one","value":1},

{"_key":"two","value":2},

{"_key":"foo","value":"bar"},

...

]

Thisformatallowslinebreakswithintheinputfileasrequired.Thedownsideisthatthewholeinputfilewillneedtobereadbyarangoimpbeforeitcansendthefirstbatch.Thismightbeaproblemiftheinputfileisbig.Bydefault,arangoimpwillallowimportingsuchfilesuptoasizeofabout16MB.

Ifyouwanttoallowyourarangoimpinstancetousemorememory,youmaywanttoincreasethemaximumfilesizebyspecifyingthecommand-lineoption--batch-size.Forexample,tosetthebatchsizeto32MB,usethefollowingcommand:

>arangoimp--file"data.json"--typejson--collection"users"--batch-size33554432

Pleasealsonotethatyoumayneedtoincreasethevalueof--batch-sizeifasingledocumentinsidetheinputfileisbiggerthanthevalueof--batch-size.

ImportingCSVData

Arangoimp

397

http://jsonlines.org

arangoimpalsooffersthepossibilitytoimportdatafromCSVfiles.ThiscomeshandywhenthedataathandisinCSVformatalreadyandyoudon'twanttospendtimeconvertingthemtoJSONfortheimport.

ToimportdatafromaCSVfile,makesureyourfilecontainstheattributenamesinthefirstrow.Allthefollowinglinesinthefilewillbeinterpretedasdatarecordsandwillbeimported.

TheCSVimportrequiresthedatatohaveahomogeneousstructure.Allrecordsmusthaveexactlythesameamountofcolumnsasthereareheaders.Bydefault,lineswithadifferentnumberofvalueswillnotbeimportedandtherewillbewarningsforthem.Tostillimportlineswithlessvaluesthanintheheader,thereisthe--ignore-missingoption.Ifsettotrue,linesthathaveadifferentamountoffieldswillbeimported.Inthiscaseonlythoseattributeswillbepopulatedforwhichtherearevalues.Attributesforwhichtherearenovaluespresentwillsilentlybediscarded.

Example:

"first","last","age","active","dob"

"John","Connor",25,true

"Jim","O'Brady"

With--ignore-missingthiswillproducethefollowingdocuments:

{"first":"John","last":"Connor","active":true,"age":25}

{"first":"Jim","last":"O'Brady"}

Thecellvaluescanhavedifferentdatatypesthough.Ifacelldoesnothaveanyvalue,itcanbeleftemptyinthefile.Thesevalueswillnotbeimportedsotheattributeswillnot"bethere"indocumentcreated.Valuesenclosedinquoteswillbeimportedasstrings,sotoimportnumericvalues,booleanvaluesorthenullvalue,don'tenclosethevalueinquotesinyourfile.

We'llbeusingthefollowingimportfortheCSVimport:

"first","last","age","active","dob"

"John","Connor",25,true,

"Jim","O'Brady",19,,

"Lisa","Jones",,,"1981-04-09"

Hans,dosSantos,0123,,

Wayne,Brewer,,false,

Thecommandlinetoexecutetheimportis:

>arangoimp--file"data.csv"--typecsv--collection"users"

Theabovedatawillbeimportedinto5documentswhichwilllookasfollows:

{"first":"John","last":"Connor","active":true,"age":25}

{"first":"Jim","last":"O'Brady","age":19}

{"first":"Lisa","last":"Jones","dob":"1981-04-09"}

{"first":"Hans","last":"dosSantos","age":123}

{"first":"Wayne","last":"Brewer","active":false}

Ascanbeseen,valuesleftcompletelyemptyintheinputfilewillbetreatedasabsent.Numericvaluesnotenclosedinquoteswillbetreatedasnumbers.Notethatleadingzerosinnumericvalueswillberemoved.Toimportnumberswithleadingzeros,pleaseusestrings.Theliteralstrueandfalsewillbetreatedasbooleansiftheyarenotenclosedinquotes.Othervaluesnotenclosedinquoteswillbetreatedasstrings.Anyvaluesenclosedinquoteswillbetreatedasstrings,too.

Stringvaluescontainingthequotecharacterortheseparatormustbeenclosedwithquotecharacters.Withinastring,thequotecharacteritselfmustbeescapedwithanotherquotecharacter(orwithabackslashifthe--backslash-escapeoptionisused).

Notethatthequoteandseparatorcharacterscanbeadjustedviathe--quoteand--separatorargumentswheninvokingarangoimp.Thequotecharacterdefaultstothedoublequote(").Tousealiteralquoteinastring,youcanusetwoquotecharacters.Tousebackslashforescapingquotecharacters,pleasesettheoption--backslash-escapetotrue.

TheimportersupportsWindows(CRLF)andUnix(LF)linebreaks.Linebreaksmightalsooccurinsidevaluesthatareenclosedwiththequotecharacter.

Arangoimp

398

Here'sanexampleforusingliteralquotesandnewlinesinsidevalues:

"name","password"

"Foo","r4ndom""123!"

"Bar","wow!

thisisa

multinepassword!"

"Bartholomew""Bart""Simpson","Milhouse"

Extrawhitespaceattheendofeachlinewillbeignored.Whitespaceatthestartoflinesorbetweenfieldvalueswillnotbeignored,sopleasemakesurethatthereisnoextrawhitespaceinfrontofvaluesorbetweenthem.

ImportingTSVData

Youmayalsoimporttab-separatedvalues(TSV)fromafile.Thisformatisverysimple:everylineinthefilerepresentsadatarecord.Thereisnoquotingorescaping.Thatalsomeansthattheseparatorcharacter(whichdefaultstothetabstopsymbol)mustnotbeusedanywhereintheactualdata.

AswithCSV,thefirstlineintheTSVfilemustcontaintheattributenames,andalllinesmusthaveanidenticalnumberofvalues.

Ifadifferentseparatorcharacterorstringshouldbeused,itcanbespecifiedwiththe--separatorargument.

AnexamplecommandlinetoexecutetheTSVimportis:

>arangoimp--file"data.tsv"--typetsv--collection"users"

AttributeNameTranslation

FortheCSVandTSVinputformats,attributenamescanbetranslatedautomatically.ThisisusefulincasetheimportfilehasdifferentattributenamesthanthosethatshouldbeusedinArangoDB.

Acommonusecaseistorenamean"id"columnfromtheinputfileinto"_key"asitisexpectedbyArangoDB.Todothis,specifythefollowingtranslationwheninvokingarangoimp:

>arangoimp--file"data.csv"--typecsv--translate"id=_key"

Othercommoncasesaretorenamecolumnsintheinputfileto_fromand_to:

>arangoimp--file"data.csv"--typecsv--translate"from=_from"--translate"to=_to"

Thetranslateoptioncanbespecifiedmultipletypes.Thesourceattributenameandthetargetattributemustbeseparatedwitha=.

IgnoringAttributes

FortheCSVandTSVinputformats,certainattributenamescanbeignoredonimports.InanArangoDBclustertherearecaseswherethiscancomeinhandy,whenyourdocumentsalreadycontaina_keyattributeandyourcollectionhasashardingattributeotherthan_key:Intheclusterthisconfigurationisnotsupported,becauseArangoDBneedstoguaranteetheuniquenessofthe_keyattributeinallshardsofthecollection.

>arangoimp--file"data.csv"--typecsv--remove-attribute"_key"

Thesamethingwouldapplyifyourdatacontainsan_idattribute:

>arangoimp--file"data.csv"--typecsv--remove-attribute"_id"

ImportingintoanEdgeCollection

Arangoimp

399

arangoimpcanalsobeusedtoimportdataintoanexistingedgecollection.Theimportdatamust,foreachedgetoimport,containatleastthe_fromand_toattributes.Theseindicatewhichothertwodocumentstheedgeshouldconnect.Itisnecessarythattheseattributesaresetforallrecords,andpointtovaliddocumentidsinexistingcollections.

Examples

{"_from":"users/1234","_to":"users/4321","desc":"1234isconnectedto4321"}

Note:Theedgecollectionmustalreadyexistwhentheimportisstarted.Usingthe--create-collectionflagwillnotworkbecausearangoimpwillalwaystrytocreatearegulardocumentcollectionifthetargetcollectiondoesnotexist.

Updatingexistingdocuments

Bydefault,arangoimpwilltrytoinsertalldocumentsfromtheimportfileintothespecifiedcollection.Incasetheimportfilecontainsdocumentsthatarealreadypresentinthetargetcollection(matchingisdoneviathe_keyattributes),thenadefaultarangoimprunwillnotimportthesedocumentsandcomplainaboutuniquekeyconstraintviolations.

However,arangoimpcanbeusedtoupdateorreplaceexistingdocumentsincasetheyalreadyexistinthetargetcollection.Itprovidesthecommand-lineoption--on-duplicatetocontrolthebehaviorincaseadocumentisalreadypresentinthedatabase.

Thedefaultvalueof--on-duplicateiserror.Thismeansthatwhentheimportfilecontainsadocumentthatispresentinthetargetcollectionalready,thentryingtore-insertadocumentwiththesame_keyvalueisconsideredanerror,andthedocumentinthedatabasewillnotbemodified.

Otherpossiblevaluesfor--on-duplicateare:

update:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbeupdatedbyarangoimp.updatewillperformapartialupdateoftheexistingdocument,modifyingonlytheattributesthatarepresentintheimportfileandleavingallotherattributesuntouched.

Thevaluesofsystemattributes_id,_key,_rev,_fromand_tocannotbeupdatedorreplacedinexistingdocuments.

replace:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbereplacebyarangoimp.replacewillreplacetheexistingdocumententirely,resultinginadocumentwithonlytheattributesspecifiedintheimportfile.

Thevaluesofsystemattributes_id,_key,_rev,_fromand_tocannotbeupdatedorreplacedinexistingdocuments.

ignore:eachdocumentpresentintheimportfilethatisalsopresentinthetargetcollectionalreadywillbeignoredandnotmodifiedinthetargetcollection.

When--on-duplicateissettoeitherupdateorreplace,arangoimpwillreturnthenumberofdocumentsupdated/replacedintheupdatedreturnvalue.Whensettoanothervalue,thevalueofupdatedwillalwaysbezero.When--on-duplicateissettoignore,arangoimpwillreturnthenumberofignoreddocumentsintheignoredreturnvalue.Whensettoanothervalue,ignoredwillalwaysbezero.

Itispossibletoperformacombinationofinsertsandupdates/replaceswithasinglearangoimprun.When--on-duplicateissettoupdateorreplace,alldocumentspresentintheimportfilewillbeinsertedintothetargetcollectionprovidedtheyarevalidanddonotalreadyexistwiththespecified_key.Documentsthatarealreadypresentinthetargetcollection(identifiedby_keyattribute)willinsteadbeupdated/replaced.

Arangoimpresultoutput

Anarangoimpimportrunwillprintoutthefinalresultsonthecommandline.Itwillshowthe

numberofdocumentscreated(created)numberofdocumentsupdated/replaced(updated/replaced,onlynon-zeroif--on-duplicatewassettoupdateorreplace,seebelow)numberofwarningsorerrorsthatoccurredontheserverside(warnings/errors)numberofignoreddocuments(onlynon-zeroif--on-duplicatewassettoignore).

Example

created:2

warnings/errors:0

updated/replaced:0

ignored:0

Arangoimp

400

ForCSVandTSVimports,thetotalnumberofinputfilelinesreadwillalsobeprinted(linesread).

arangoimpwillalsoprintoutdetailsaboutwarningsanderrorsthathappenedontheserver-side(ifany).

AttributeNamingandSpecialAttributes

AttributeswhosenamesstartwithanunderscorearetreatedinaspecialwaybyArangoDB:

theoptional_keyattributecontainsthedocument'skey.Ifspecified,thevaluemustbeformallyvalid(e.g.mustbeastringandconformtothenamingconventions).Additionally,thekeyvaluemustbeuniquewithinthecollectiontheimportisrunfor._from:whenimportingintoanedgecollection,thisattributecontainstheidofoneofthedocumentsconnectedbytheedge.Thevalueof_frommustbeasyntacticallyvaliddocumentidandthereferredcollectionmustexist._to:whenimportingintoanedgecollection,thisattributecontainstheidoftheotherdocumentconnectedbytheedge.Thevalueof_tomustbeasyntacticallyvaliddocumentidandthereferredcollectionmustexist._rev:thisattributecontainstherevisionnumberofadocument.However,therevisionnumbersaremanagedbyArangoDBandcannotbespecifiedonimport.Thusanyvalueinthisattributeisignoredonimport.

Ifyouimportvaluesinto_key,youshouldmakesuretheyarevalidandunique.

Whenimportingdataintoanedgecollection,youshouldmakesurethatallimportdocumentscan_fromand_toandthattheirvaluespointtoexistingdocuments.

Toavoidspecifyingcompletedocumentids(consistingofcollectionnamesanddocumentkeys)for_fromand_tovalues,therearetheoptions--from-collection-prefixand--to-collection-prefix.Ifspecified,thesevalueswillbeautomaticallyprependedtoeachvaluein_from(or_toresp.).Thisallowsspecifyingonlydocumentkeysinside_fromand/or_to.

Example

>arangoimp--from-collection-prefixusers--to-collection-prefixproducts...

Importingthefollowingdocumentwillthencreateanedgebetweenusers/1234andproducts/4321:

{"_from":"1234","_to":"4321","desc":"users/1234isconnectedtoproducts/4321"}

Arangoimp

401

DumpingDatafromanArangoDBdatabaseTodumpdatafromanArangoDBserverinstance,youwillneedtoinvokearangodump.Dumpscanbere-importedwitharangorestore.arangodumpcanbeinvokedbyexecutingthefollowingcommand:

unix>arangodump--output-directory"dump"

ThiswillconnecttoanArangoDBserveranddumpallnon-systemcollectionsfromthedefaultdatabase(_system)intoanoutputdirectorynameddump.Invokingarangodumpwillfailiftheoutputdirectoryalreadyexists.Thisisanintentionalsecuritymeasuretopreventyoufromaccidentallyoverwritingalreadydumpeddata.Ifyouarepositivethatyouwanttooverwritedataintheoutputdirectory,youcanusetheparameter--overwritetruetoconfirmthis:

unix>arangodump--output-directory"dump"--overwritetrue

arangodumpwillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:

--server.database:nameofthedatabasetoconnectto--server.endpoint:endpointtoconnectto--server.username:username--server.password:passwordtouse(omitthisandyou'llbepromptedforthepassword)--server.authentication:whetherornottouseauthentication

Here'sanexampleofdumpingdatafromanon-standardendpoint,usingadedicateddatabasename:

unix>arangodump--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--output-directory

"dump"

Whenfinished,arangodumpwillprintoutasummarylinewithsomeaggregatestatisticsaboutwhatitdid,e.g.:

Processed43collection(s),wrote408173500byte(s)intodatafiles,sent88batch(es)

Bydefault,arangodumpwilldumpbothstructuralinformationanddocumentsfromallnon-systemcollections.Toadjustthis,therearethefollowingcommand-linearguments:

--dump-data:settotruetoincludedocumentsinthedump.Settofalsetoexcludedocuments.Thedefaultvalueistrue.--include-system-collections:whetherornottoincludesystemcollectionsinthedump.Thedefaultvalueisfalse.

Forexample,toonlydumpstructuralinformationofallcollections(includingsystemcollections),use:

unix>arangodump--dump-datafalse--include-system-collectionstrue--output-directory"dump"

Torestrictthedumptojustspecificcollections,thereisisthe--collectionoption.Itcanbespecifiedmultipletimesifrequired:

unix>arangodump--collectionmyusers--collectionmyvalues--output-directory"dump"

Structuralinformationforacollectionwillbesavedinfileswithnamepattern.structure.json.EachstructurefilewillcontainsaJSONobjectwiththeseattributes:

parameters:containsthecollectionpropertiesindexes:containsthecollectionindexes

Documentdataforacollectionwillbesavedinfileswithnamepattern.data.json.Eachlineinadatafileisadocumentinsertion/updateordeletionmarker,alongsidewithsomemetadata.

StartingwithVersion2.1ofArangoDB,thearangodumptoolalsosupportssharding.Simplypointittooneofthecoordinatorsanditwillbehaveexactlyasdescribedabove,workingonshardedcollectionsinthecluster.

Arangodump

402

However,asopposedtothesingleinstancesituation,thisoperationdoesnotguaranteetodumpaconsistentsnapshotifwriteoperationshappenduringthedumpoperation.Itisthereforerecommendednottoperformanydata-modifcationoperationsontheclusterwhilstarangodumpisrunning.

Asabove,theoutputwillbeonestructuredescriptionfileandonedatafilepershardedcollection.Notethatthedatainthedatafileissortedfirstbyshardsandwithineachshardbyascendingtimestamp.Thestructuralinformationofthecollectioncontainsthenumberofshardsandtheshardkeys.

NotethattheversionofthearangodumpclienttoolneedstomatchtheversionoftheArangoDBserveritconnectsto.

Advancedclusteroptions

Startingwithversion3.1.17,collectionsmaybecreatedwithsharddistributionidenticaltoanexistingprototypicalcollection;i.e.shardsaredistributedintheverysamepatternasintheprototypecollection.Suchcollectionscannotbedumpedwithoutthereferencecollectionorarangodumpwithyieldanerror.

unix>arangodump--collectionclonedCollection--output-directory"dump"

ERRORCollectionclonedCollection'ssharddistributionisbasedonathatofcollectionprototypeCollection,whichisnotdumpe

dalong.Youmaydumpthecollectionregardlessofthemissingprototypecollectionbyusingthe--ignore-distribute-shards-lik

e-errorsparameter.

Therearetwowaystoapproachthatproblem:Solveit,i.e.dumptheprototypecollectionalong:

unix>arangodump--collectionclonedCollection--collectionprototypeCollection--output-directory"dump"


Oroverridethatbehaviourtobeabletodumpthecollectionindividually.

unix>arangodump--collectionBclonedCollection--output-directory"dump"--ignore-distribute-shards-like-errors


Nothatinconsequence,restoringsuchacollectionwithoutitsprototypeisaffected.arangorestore

Encryption

IntheArangoDBEnterpriseEditiontherearetheadditionalparameters:

Encryptionkeystoredinfile

--encryption.keyfilepath-of-keyfile

Thefilepath-to-keyfilemustcontaintheencryptionkey.Thisfilemustbesecured,sothatonlyarangodcanaccessit.Youshouldalsoensurethatincasesome-onestealsthehardware,hewillnotbeabletoreadthefile.Forexample,byencryption/mytmpfsorcreatingain-memoryfile-systemunder/mytmpfs.

Encryptionkeygeneratedbyaprogram

--encryption.key-generatorpath-to-my-generator

Theprogrampath-to-my-generatormustoutputtheencryptiononstandardoutputandexit.

Creatingkeys

Theencryptionkeyfilemustcontain32bytesofrandomdata.

Youcancreateitwithacommandlinethis.

ddif=/dev/randombs=1count=32of=yourSecretKeyFile

Arangodump

403

Forsecurity,itisbesttocreatethesekeysoffline(awayfromyourdatabaseservers)anddirectlystoretheminyousecretmanagementtool.

Arangodump

404

ArangorestoreToreloaddatafromadumppreviouslycreatedwitharangodump,ArangoDBprovidesthearangorestoretool.

Pleasenotethatarangorestoremustnotbeusedtocreateseveralsimilardatabaseinstancesinoneinstallation.

Thismeansifyouhaveanarangodumpoutputofdatabasea,andyoucreateaseconddatabasebonthesameinstanceofArangoDB,andrestorethedumpofaintob-dataintegritycannotbeguaranteed.

ReloadingDataintoanArangoDBdatabase

Invokingarangorestore

arangorestorecanbeinvokedfromthecommand-lineasfollows:

unix>arangorestore--input-directory"dump"

ThiswillconnecttoanArangoDBserverandreloadstructuralinformationanddocumentsfoundintheinputdirectorydump.Pleasenotethattheinputdirectorymusthavebeencreatedbyrunningarangodumpbefore.

arangorestorewillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:


Sinceversion2.6arangorestoreprovidestheoption--create-database.Settingthisoptiontotruewillcreatethetargetdatabaseifitdoesnotexist.Whencreatingthetargetdatabase,theusernameandpasswordspassedtoarangorestore(inoptions--server.usernameand--server.password)willbeusedtocreateaninitialuserforthenewdatabase.

Theoption--force-same-databaseallowsrestrictingarangorestoreoperationstoadatabasewiththesamenameasinthesourcedump's"dump.json"file.Itcanthusbeusedtopreventrestoringdataintoa"wrong"databasebyaccident.

Forexample,ifadumpwastakenfromdatabasea,andtherestoreisattemptedintodatabaseb,thenwiththe--force-same-databaseoptionsettotrue,arangorestorewillabortinstantly.

The--force-same-databaseoptionissettofalsebydefaulttoensurebackwards-compatibility.

Here'sanexampleofreloadingdatatoanon-standardendpoint,usingadedicateddatabasename:

unix>arangorestore--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--input-directo

ry"dump"

Tocreatethetargetdatabasewherestoring,useacommandlikethis:

unix>arangorestore--server.usernamebackup--server.databasenewdb--create-databasetrue--input-directory"dump"

arangorestorewillprintoutitsprogresswhilerunning,andwillendwithalineshowingsomeaggregatestatistics:

Processed2collection(s),read2256byte(s)fromdatafiles,sent2batch(es)

Bydefault,arangorestorewillre-createallnon-systemcollectionsfoundintheinputdirectoryandloaddataintothem.Ifthetargetdatabasealreadycontainscollectionswhicharealsopresentintheinputdirectory,theexistingcollectionsinthedatabasewillbedroppedandre-createdwiththedatafoundintheinputdirectory.

Arangorestore

405

Thefollowingparametersareavailabletoadjustthisbehavior:

--create-collection:settotruetocreatecollectionsinthetargetdatabase.Ifthetargetdatabasealreadycontainsacollectionwiththesamename,itwillbedroppedfirstandthenre-createdwiththepropertiesfoundintheinputdirectory.Settofalsetokeepexistingcollectionsinthetargetdatabase.Ifsettofalseandarangorestoreencountersacollectionthatispresentinboththetargetdatabaseandtheinputdirectory,itwillabort.Thedefaultvalueistrue.--import-data:settotruetoloaddocumentdataintothecollectionsinthetargetdatabase.Settofalsetonotloadanydocumentdata.Thedefaultvalueistrue.--include-system-collections:whetherornottoincludesystemcollectionswhenre-creatingcollectionsorreloadingdata.Thedefaultvalueisfalse.

Forexample,to(re-)createallnon-systemcollectionsandloaddocumentdataintothem,use:

unix>arangorestore--create-collectiontrue--import-datatrue--input-directory"dump"

Thiswilldroppotentiallyexistingcollectionsinthetargetdatabasethatarealsopresentintheinputdirectory.

Toincludesystemcollectionstoo,use--include-system-collectionstrue:

unix>arangorestore--create-collectiontrue--import-datatrue--include-system-collectionstrue--input-directory"dump"

To(re-)createallnon-systemcollectionswithoutloadingdocumentdata,use:

unix>arangorestore--create-collectiontrue--import-datafalse--input-directory"dump"

Thiswillalsodropexistingcollectionsinthetargetdatabasethatarealsopresentintheinputdirectory.

Tojustloaddocumentdataintoallnon-systemcollections,use:

unix>arangorestore--create-collectionfalse--import-datatrue--input-directory"dump"

Torestrictreloadingtojustspecificcollections,thereisisthe--collectionoption.Itcanbespecifiedmultipletimesifrequired:

unix>arangorestore--collectionmyusers--collectionmyvalues--input-directory"dump"

Collectionswillbeprocessedbyinalphabeticalorderbyarangorestore,withalldocumentcollectionsbeingprocessedbeforealledgecollections.Thisistoensurethatreloadingdataintoedgecollectionswillhavethedocumentcollectionslinkedinedges(_fromand_toattributes)loaded.

Encryption

Seearangodumpfordetails.

RestoringRevisionIdsandCollectionIds

arangorestorewillreloaddocumentandedgesdatawiththeexactsame_key,_fromand_tovaluesfoundintheinputdirectory.However,whenloadingdocumentdata,itwillassignitsownvaluesforthe_revattributeofthereloadeddocuments.Thoughthisdifferenceisintentional(normally,everyservershouldcreateitsown_revvalues)theremightbesituationswhenitisrequiredtore-usetheexactsame_revvaluesforthereloadeddata.Thiscanbeachievedbysettingthe--recycle-idsparametertotrue:

unix>arangorestore--collectionmyusers--collectionmyvalues--input-directory"dump"

Notethatsetting--recycle-idstotruewillalsocausecollectionstobe(re-)createdinthetargetdatabasewiththeexactsamecollectionidasintheinputdirectory.Anypotentiallyexistingcollectioninthetargetdatabasewiththesamecollectionidwillthenbedropped.

ReloadingDataintoadifferentCollection

Arangorestore

406

Withsomecreativityyoucanusearangodumpandarangorestoretotransferdatafromonecollectionintoanother(eitheronthesameserverornot).Forexample,tocopydatafromacollectionmyvaluesindatabasemydbintoacollectionmycopyvaluesindatabasemycopy,youcanstartwiththefollowingcommand:

unix>arangodump--collectionmyvalues--server.databasemydb--output-directory"dump"

Thiswillcreatetwofiles,myvalues.structure.jsonandmyvalues.data.json,intheoutputdirectory.Toloaddatafromthedatafileintoanexistingcollectionmycopyvaluesindatabasemycopy,renamethefilestomycopyvalues.structure.jsonandmycopyvalues.data.json.Afterthat,runthefollowingcommand:

unix>arangorestore--collectionmycopyvalues--server.databasemycopy--input-directory"dump"

Usingarangorestorewithsharding

AsofVersion2.1thearangorestoretoolsupportssharding.Simplypointittooneofthecoordinatorsinyourclusteranditwillworkasusualbutonshardedcollectionsinthecluster.

Ifarangorestoreisaskedtodropandre-createacollection,itwillusethesamenumberofshardsandthesameshardkeysaswhenthecollectionwasdumped.Thedistributionoftheshardstotheserverswillalsobethesameasatthetimeofthedump.ThismeansinparticularthatDBserverswiththesameIDsasbeforemustbepresentintheclusterattimeoftherestore.

Ifacollectionwasdumpedfromasingleinstance,onecanmanuallyaddthestructuraldescriptionfortheshardkeysandthenumberanddistributionoftheshardsandthentherestoreintoaclusterwillwork.

IfyourestoreacollectionthatwasdumpedfromaclusterintoasingleArangoDBinstance,thenumberofshardsandtheshardkeyswillsilentlybeignored.

Notethatinacluster,everynewlycreatedcollectionwillhaveanewID,itisnotpossibletoreusetheIDfromtheoriginallydumpedcollection.ThisisforsafetyreasonstoensureconsistencyofIDs.

Restoringcollectionswithshardingprototypes

arangorestorewillyieldanerror,whiletryingtorestoreacollection,whosesharddistributionfollowsacollection,whichdoesnotexistintheclusterandwhichwasnotdumpedalong:

unix>arangorestore--collectionclonedCollection--server.databasemydb--input-directory"dump"

ERRORgoterrorfromserver:HTTP500(InternalServerError):ArangoError1486:mustnothaveadistributeShardsLikeattribute

pointingtoanunknowncollection

Processed0collection(s),read0byte(s)fromdatafiles,sent0batch(es)

Thecollectioncanberestoredbyoverridingtheerrormessageasfollows:

unix>arangorestore--collectionclonedCollection--server.databasemydb--input-directory"dump"--ignore-distribute-shards-li

ke-errors

RestoreintoanauthenticationenabledArangoDB

OfcourseyoucanrestoredataintoapasswordprotectedArangoDBaswell.Howeverthisrequirescertainuserrightsfortheuserusedintherestoreprocess.TherightsaredescribedindetailintheManagingUserschapter.Forrestorethisshortoverviewissufficient:

Whenimportingintoanexistingdatabase,thegivenuserneedsAdministrateaccessonthisdatabase.WhencreatinganewDatabaseduringrestore,thegivenuserneedsAdministrateaccesson_system.TheuserwillbepromotedwithAdministrateaccessonthenewlycreateddatabase.

Arangorestore

407

ExportingDatafromanArangoDBdatabaseToexportdatafromanArangoDBserverinstance,youwillneedtoinvokearangoexport.arangoexportcanbeinvokedbyexecutingthefollowingcommand:

unix>arangoexport--collectiontest--output-directory"dump"

Thisexportsthecollectionstestintothedirectorydumpasonebigjsonarray.Everyentryinthisarrayisonedocumentfromthecollectionwithoutaspecificorder.Toexportmorethanonecollectionatatimespecifymultiple--collectionoptions.

Thedefaultoutputdirectoryisexport.

arangoexportwillbydefaultconnecttothe_systemdatabaseusingthedefaultendpoint.Ifyouwanttoconnecttoadifferentdatabaseoradifferentendpoint,oruseauthentication,youcanusethefollowingcommand-lineoptions:


Here'sanexampleofexportingdatafromanon-standardendpoint,usingadedicateddatabasename:

unix>arangoexport--server.endpointtcp://192.168.173.13:8531--server.usernamebackup--server.databasemydb--collectiontes

t--output-directory"my-export"

Whenfinished,arangoexportwillprintoutasummarylinewithsomeaggregatestatisticsaboutwhatitdid,e.g.:

Processed2collection(s),wrote9031763Byte(s),78HTTPrequest(s)

ExportJSON

unix>arangoexport--typejson--collectiontest

Thisexportsthecollectiontestintotheoutputdirectoryexportasonejsonarray.Everyarrayentryisonedocumentfromthecollectiontest

ExportJSONL

unix>arangoexport--typejsonl--collectiontest

Thisexportsthecollectiontestintotheoutputdirectoryexportasjsonl.Everylineintheexportisonedocumentfromthecollectiontestasjson.

ExportCSV

unix>arangoexport--typecsv--collectiontest--fields_key,_id,_rev

ThisexportsthecollectiontestintotheoutputdirectoryexportasCSV.Thefirstlinecontainstheheaderwithallfieldnames.EachlineisonedocumentrepresentedasCSVandseparatedwithacomma.ObjectsandArraysarerepresentedasaJSONstring.

Arangoexport

408

http://jsonlines.org

ExportXML

unix>arangoexport--typexml--collectiontest

ThisexportsthecollectiontestintotheoutputdirectoryexportasgenericXML.TherootelementofthegeneratedXMLfileisnamedcollection.EachdocumentinthecollectionisexportedinadocXMLattribute.Eachdocumentattributeisexportinagenericattelement,whichhasatypeattributeindicatingtheattributevalue,andavalueattributecontainingtheattribute'svalue.

ExportXGMMLXGMMLisanXMLapplicationbasedonGML.ToviewtheXGMMLfileyoucanuseforexampleCytoscape.

importantnoteIfyouexportallattributes(--xgmml-label-onlyfalse)keepinmindthataatrributenamestypehavetobethesametypeforalldocuments.Itwontworkifyouhaveaattributenamedrankthatisinonedocumentastringandinanotherdocumentainteger.

Bad

//doc1

{

"rank":1

}

//doc2

{

"rank":"2"

}

Good

//doc1

{

"rank":1

}

//doc2

{

"rank":2

}

XGMMLspecificoptions--xgmml-label-attributespecifythenameoftheattributethatwillbecomethelabelinthexgmmlfile.

--xgmml-label-onlysettotruewillonlyexportthelabelwithoutanyattributesinedgesornodes.

exportbasedoncollections

unix>arangoexport--typexgmml--graph-namemygraph--collectionvertex--collectionedge

Thisexportstheaunnamedgraphwithvertexcollectionvertexandedgecollectionedgeintothexgmmlfilemygraph.xgmml.

exportbasedonanamedgraph

unix>arangoexport--typexgmml--graph-namemygraph

Arangoexport

409

https://en.wikipedia.org/wiki/XGMML

https://en.wikipedia.org/wiki/Graph_Modelling_Language

http://cytoscape.org

Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmml.

exportXGMMLwithoutattributes

unix>arangoexport--typexgmml--graph-namemygraph--xgmml-label-onlytrue

Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmmlwithoutthe<att>taginnodesandedges.

exportXGMMLwithaspecificlabel

unix>arangoexport--typexgmml--graph-namemygraph--xgmml-label-attributename

Thisexportsthenamedgraphmygraphintothexgmmlfilemygraph.xgmmlwithalabelfromdocumentsattributenameinsteadofthedefaultattributelabel.

ExportviaAQLquery

unix>arangoexport--typejsonl--query"forbookinbooksfilterbook.sells>100returnbook"

Exportviaanaqlqueryallowsyoutoexportthereturneddataasthetypespecifiedwith--type.Theexampleexportsallbooksasjsonlthataresoldmorethan100times.

Arangoexport

410

ManagingUsersTheusermanagementinArangoDB3issimilartotheonesfoundinMySQL,PostgreSQL,orotherdatabasesystems.

Usermanagementispossibleinthewebinterfaceandinarangoshwhileloggedontothe_systemdatabase.

Notethattheonlyusernamesmustnotstartwith:role:.

ActionsandAccessLevelsAnArangoDBservercontainsalistofusers.Italsodefinesvariousaccesslevelsthatcanbeassignedtoauser(fordetails,seebelow)andthatareneededtoperformcertainactions.Theseactionscanbegroupedintothreecategories:

serveractionsdatabaseactionscollectionactions

Theserveractionsare

createuser:allowstocreateanewuser.

updateuser:allowstochangetheaccesslevelsanddetailsofanexistinguser.

dropuser:allowstodeleteanexistinguser.

createdatabase:allowstocreateanewdatabase.

dropdatabase:allowstodeleteanexistingdatabase.

shutdownserver:removeserverfromclusterandshutdown

Thedatabaseactionsaretiedtoagivendatabase,andaccesslevelsmustbesetforeachdatabaseindividually.Foragivendatabasetheactionsare

createcollection:allowstocreateanewcollectioninthegivendatabase.

updatecollection:allowstoupdatepropertiesofanexistingcollection.

dropcollection:allowstodeleteanexistingcollection.

createindex:allowstocreateanindexforanexistingcollectioninthegivendatabase.

dropindex:allowstodeleteanindexofanexistingcollectioninthegivendatabase.

Thecollectionactionsaretiedtoagivencollectionofagivendatabase,andaccesslevelsmustbesetforeachcollectionindividually.Foragivencollectiontheactionsare

readdocument:readadocumentofthegivencollection.

createdocument:createsanewdocumentinthegivencollection.

modifydocument:modifiesanexistingdocumentofthegivencollection,thiscanbeanupdateorreplaceoperation.

dropdocument:deletesanexistingdocumentofthegivencollection.

truncatecollection:deletesalldocumentsofagivencollection.

Toperformactionsontheserverleveltheuserneedsatleastthefollowingaccesslevels.TheaccesslevelsareAdministrateandNoaccess:

serveraction serverlevel

createadatabase Administrate

dropadatabase Administrate

createauser Administrate

ManagingUsers

411

updateauser Administrate

updateuseraccesslevel Administrate

dropauser Administrate

shutdownserver Administrate

Toperformactionsinaspecificdatabase(likecreatingordroppingcollections),auserneedsatleastthefollowingaccesslevel.ThepossibleaccesslevelsfordatabasesareAdministrate,AccessandNoaccess.TheaccesslevelsforcollectionsareRead/Write,ReadOnlyandNoAccess.

databaseaction databaselevel collectionlevel

createcollection Administrate Read/Write

listcollections Access ReadOnly

renamecollection Administrate Read/Write

modifycollectionproperties Administrate Read/Write

readproperties Access ReadOnly

dropcollection Administrate Read/Write

createanindex Administrate Read/Write

dropanindex Administrate Read/Write

seeindexdefinition Access ReadOnly

NotethattheaccesslevelAccessforadatabaseisalwaysrequiredtoperformanyactiononacollectioninthatdatabase.

Forcollectionsauserneedsthefollowingaccesslevelstothegivendatabaseandthegivencollection.TheaccesslevelsforthedatabaseareAdministrate,AccessandNoaccess.TheaccesslevelsforthecollectionareRead/Write,ReadOnlyandNoAccess.

action collectionlevel databaselevel

readadocument Read/WriteorReadOnly AdministrateorAccess

createadocument Read/Write AdministrateorAccess

modifyadocument Read/Write AdministrateorAccess

dropadocument Read/Write AdministrateorAccess

truncateacollection Read/Write AdministrateorAccess

Example

Forexample,given

adatabaseexampleacollectiondatainthedatabaseexampleauserJohnSmith

IftheuserJohnSmithisassignedtheaccesslevelAccessforthedatabaseexampleandthelevelRead/Writeforthecollectiondata,thentheuserisallowedtoread,create,modifyordeletedocumentsinthecollectiondata.Buttheuseris,forexample,notallowedtocreateindexesforthecollectiondatanorcreatenewcollectionsinthedatabaseexample.

GrantingAccessLevels

Accesslevelscanbemanagedviathewebinterfaceorinarangosh.

Inordertograntanaccessleveltoauser,youcanassignoneofthreeaccesslevelsforeachdatabaseandoneofthreelevelsforeachcollectioninadatabase.Theserveraccesslevelfortheuserfollowsfromthedatabaseaccesslevelinthe_systemdatabase,itisAdministrateifandonlyifthedatabaseaccesslevelisAdministrate.NotethatthismeansthatdatabaseaccesslevelAccessdoesnotgrantauserserveraccesslevelAdministrate.

ManagingUsers

412

InitialAccessLevels

WhenausercreatesadatabasetheaccessleveloftheuserforthatdatabaseissettoAdministrate.Thesameistrueforcreatingacollection,inthiscasetheusergetRead/Writeaccesstothecollection.

WildcardDatabaseAccessLevel

Withtheabovedefinition,onemustdefinethedatabaseaccesslevelforalldatabase/userpairsintheserver,whichwouldbeverytedious.Inordertosimplifythisprocess,itispossibletodefine,forauser,awildcarddatabaseaccesslevel.Thiswildcardisusedifthedatabaseaccesslevelisnotexplicitlydefinedforacertaindatabase.EachnewcreateduserhasaninitialdatabasewildcardofNoAccess.

Changingthewildcarddatabaseaccesslevelforauserwillchangetheaccesslevelforalldatabasesthathavenoexplicitlydefinedaccesslevel.Notethatthisincludesdatabaseswhichwillbecreatedinthefutureandforwhichnoexplicitaccesslevelsaresetforthatuser!

Ifyoudeletethewildcard,thedefaultaccesslevelisdefinedasNoAccess.

TherootuserhasaninitialdatabasewildcardofAdministrate.

Example

AssumeuserJohnSmithhasthefollowingdatabaseaccesslevels:

accesslevel

database* Access

databaseshop1 Administrate

databaseshop2 NoAccess

ThiswillgivetheuserJohnSmiththefollowingdatabaselevelaccess:

databaseshop1:Administratedatabaseshop2:NoAccessdatabasesomething:Access

Ifthewildcard*ischangedfromAccesstoNoAccessthenthepermissionswillchangeasfollows:

databaseshop1:Administratedatabaseshop2:NoAccessdatabasesomething:NoAccess

WildcardCollectionAccessLevel

Foreachuseranddatabasethereisawildcardcollectionaccesslevel.Thislevelisusedforallcollectionspairswithoutanexplicitlydefinedcollectionaccesslevel.Notethatthisincludescollectionswhichwillbecreatedinthefutureandforwhichnoexplicitaccesslevelsaresetforathatuser!EachnewcreateduserhasaninitialcollectionwildcardofNoAccess.

Ifyoudeletethewildcard,thesystemdefaultstoNoAccess.

TherootuserhasaninitialcollectionwildcardofRead/Writeineverydatabase.

Whencreatingauserthroughdb._createDatabase(name,options,users)theaccessleveloftheuserforthisdatabasewillbesettoAdministrateandthewildcardforallcollectionswithinthisdatabasewillbesettoRead/Write.

Example

AssumeuserJohnSmithhasthefollowingdatabaseaccesslevels:

accesslevel

database* Access

andcollectionaccesslevels:

accesslevel

ManagingUsers

413

database*,collection* Read/Write

databaseshop1,collectionproducts Read-Only

databaseshop1,collection* NoAccess

databaseshop2,collection* Read-Only

Thentheuserdoewillgetthefollowingcollectionaccesslevels:

databaseshop1,collectionproducts:Read-Onlydatabaseshop1,collectioncustomers:NoAccessdatabaseshop2,collectionreviews:Read-Onlydatabasesomething,collectionelse:Read/Write

Explanation:

Databaseshop1,collectionproductsdirectlymatchesadefinedaccesslevel.ThislevelisdefinedasRead-Only.

Databaseshop1,collectioncustomersdoesnotmatchadefinedaccesslevel.However,databaseshop1matchesandthewildcardinthisdatabaseforcollectionlevelisNoAccess.

Databaseshop2,collectionreviewsdoesnotmatchadefinedaccesslevel.However,databaseshop2matchesandthewildcardinthisdatabaseforcollectionlevelisRead-Only.

Databasesomehing,collectionelsedoesnotmatchadefinedaccesslevel.Thedatabasesomethingalsodoeshaveadirectmatches.Thereforethewildcardisselected.ThelevelisRead/Write.

PermissionResolution

Theaccesslevelsfordatabasesandcollectionsareresolvedinthefollowingway:

Foradatabase"foo":

1. Checkifthereisaspecificdatabasegrantforfoo,ifyesusethegrantedaccesslevel2. Chooseathehigheraccesslevelof:

Awildcarddatabasegrant(forexamplegrantDatabase('user','*','rw')Adatabasegrantonthe_systemdatabase

Foracollectionnamed"bar":

1. Checkifthereisaspecificdatabasegrantforbar,ifyesusethegrantedaccesslevel2. Chooseathehigheraccesslevelof:

Anywildcardaccessgrantinthesamedatabase,oron"/"(inthisexamplegrantCollection('user','foo','*','rw'))Theaccesslevelforthecurrentdatabase(inthisexamplegrantDatabase('user','foo','rw')Theaccesslevelforthe_systemdatabase

Anexceptiontothisaresystemcollections,hereonlytheaccesslevelforthedatabaseisused.

SystemCollections

Theaccesslevelforsystemcollectionscannotbechanged.Theyfollowdifferentrulesthanuserdefinedcollectionsandmaychangewithoutfurthernotice.Currentlythesystemcollectionsfollowtheserules:

collection accesslevel

_users(in_system) NoAccess

_queues Read-Only

_frontend Read/Write

* sameasdb

AllothersystemcollectionshaveaccesslevelRead/WriteiftheuserhasAdministrateaccesstothedatabase.TheyhaveaccesslevelRead/OnlyiftheuserhasAccesstothedatabase.

ManagingUsers

414

TomodifythesesystemcollectionsyoushouldalwaysusethespecializedAPIsprovidedbyArangoDB.Forexamplenouserhasaccesstothe_userscollectioninthe_systemdatabase.Allchangestotheaccesslevelsmustbedoneusingthe@arangodb/usersmodule,the/_users/APIorthewebinterface.

LDAPUsers

ThisfeatureisonlyavailableintheEnterpriseEdition.

ArangoDBsupportsLDAPasanexternalauthenticationsystem.FordetailedinformationpleasehavelookintotheLDAPconfigurationguide.

ThereareafewdifferencestonormalArangoDBusers:

ArangoDBdoesnot"know"LDAPusersbeforetheyfirstauthenticate,callstovariousAPI'susingendpointsin_api/users/*willfailuntiltheuserfirstlogs-inAccesslevelsofeachuserareperiodicallyupdated,thiswillhappenbydefaultevery5minutesItisnotpossibletochangepermissionsonLDAPusersdirectly,onlyonrolesLDAPuserscannotstoreconfigurationdataperuser(affectsforexamplecustomsettingsinthegraphviewer)

TograntaccessforanLDAPuseryouwillneedtocreateroleswithintheArangoDBserver.Aroleisjustauserwiththe":role:"prefixinitsname.Roleuserscannotloginasdatabaseusers,the":role:"prefixensuresthis.YourLDAPuserswillneedtohaveatleastonerole,oncetheuserlogsinhewillbeautomaticallygrantedtheunionofallaccessrightsofallhisroles.Notethatalowerrightgrantinonerolewillbeoverwrittenbyahigheraccessgrantinadifferentrole.

ManagingUsers

415

ManagingUsersintheArangoDBShellPleasenote,thatforbackwardcompatibilitytheserveraccesslevelsfollowfromthedatabaseaccesslevelonthedatabase_system.

Alsonotethattheserveranddatabaseaccesslevelsarerepresentedas

rw:forAdministratero:forAccessnone:forNoaccess

Thisisagainforbackwardcompatibility.

Example

Fireuparangoshandrequiretheusersmodule.Useittocreateanewuser:

arangosh>varusers=require('@arangodb/users');

arangosh>users.save('JohnSmith','mypassword');

CreatesausercalledJohnSmith.Thisuserwillhavenoaccessatall.

arangosh>users.grantDatabase('JohnSmith','testdb','rw');

ThisgrantstheuserAdministrateaccesstothedatabasetestdb.revokeDatabasewillrevokethisaccesslevelsetting.

Note:Beawarethatfrom3.2onwardsthegrantDatabasewillnotautomaticallygrantuserstheaccessleveltowriteorreadcollectionsinadatabase.IfyougrantaccesstoadatabasetestdbyouwilladditionallyneedtoexplicitlygrantaccesslevelstoindividualcollectionsviagrantCollection.

Theupgradeprocedurefrom3.1to3.2setsthewildcarddatabaseaccesslevelforalluserstoAdministrateandsetsthewildcardcollectionaccesslevelforalluser/databasepairstoRead/Write.

arangosh>users.grantCollection('JohnSmith','testdb','testcoll','rw');

Saveusers.save(user,passwd,active,extra)

ThiswillcreateanewArangoDBuser.Theusernamemustbespecifiedinuserandmustnotbeempty.

Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.IfyoupassthespecialvalueARANGODB_DEFAULT_ROOT_PASSWORD,thepasswordwillbesetthevaluestoredintheenvironmentvariableARANGODB_DEFAULT_ROOT_PASSWORD.ThiscanbeusedtopassaninstancevariableintoArangoDB.Forexample,theinstanceidentifierfromAmazon.

Iftheactiveattributeisnotspecified,itdefaultstotrue.Theextraattributecanbeusedtosavecustomdatawiththeuser.

Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,ortherealreadyexistsauserwiththespecifiedname.

Note:Theuserwillnothavepermissiontoaccessanydatabase.YouneedtogranttheaccessrightsforoneormoredatabasesusinggrantDatabase.

Examples

arangosh>require('@arangodb/users').save('my-user','my-secret-password');


InArangosh

416

GrantDatabaseusers.grantDatabase(user,database,type)

Thisgrantstype('rw','ro'or'none')accesstothedatabasefortheuser.Ifdatabaseis"*",thissetsthewildcarddatabaseaccesslevelfortheuseruser.

Theserveraccesslevelfollowsfromtheaccesslevelforthedatabase_system.

RevokeDatabaseusers.revokeDatabase(user,database)

Thisclearstheaccesslevelsettingtothedatabasefortheuserandthewildcarddatabaseaccesssettingforthisuserkicksin.IncasenowildcardaccesswasdefinedthedefaultisNoAccess.Thiswillalsocleartheaccesslevelsforallthecollectionsinthisdatabase.

GrantCollectionusers.grantCollection(user,database,collection,type)

Thisgrantstype('rw','ro'or'none')accessleveltothecollectionindatabasefortheuser.Ifcollectionis"*"thissetsthewildcardcollectionaccesslevelfortheuseruserindatabasedatabase.

RevokeCollectionusers.revokeCollection(user,database)

Thisclearstheaccesslevelsettingtothecollectioncollectionfortheuseruser.ThesystemwilleitherfallbacktothewildcardcollectionaccesslevelordefaulttoNoAccess

Replaceusers.replace(user,passwd,active,extra)

ThiswilllookupanexistingArangoDBuserandreplaceitsuserdata.

Theusernamemustbespecifiedinuser,andauserwiththespecifiednamemustalreadyexistinthedatabase.

Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.

Iftheactiveattributeisnotspecified,itdefaultstotrue.Theextraattributecanbeusedtosavecustomdatawiththeuser.

Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,orifthespecifiedusercannotbefoundinthedatabase.

Note:thisfunctionwillnotworkfromwithinthewebinterface

Examples

arangosh>require("@arangodb/users").replace("my-user","my-changed-password");


Updateusers.update(user,passwd,active,extra)

ThiswillupdateanexistingArangoDBuserwithanewpasswordandotherdata.

Theusernamemustbespecifiedinuserandtheusermustalreadyexistinthedatabase.

InArangosh

417

Thepasswordmustbegivenasastring,too,butcanbeleftemptyifrequired.

Iftheactiveattributeisnotspecified,thecurrentvaluesavedfortheuserwillnotbechanged.Thesameistruefortheextraattribute.

Thismethodwillfailifeithertheusernameorthepasswordsarenotspecifiedorgiveninawrongformat,orifthespecifiedusercannotbefoundinthedatabase.

Examples

arangosh>require("@arangodb/users").update("my-user","my-secret-password");


isValidusers.isValid(user,password)

Checkswhetherthegivencombinationofusernameandpasswordisvalid.Thefunctionwillreturnabooleanvalueifthecombinationofusernameandpasswordisvalid.

Eachcalltothisfunctionispenalizedbytheserversleepingarandomamountoftime.

Examples

arangosh>require("@arangodb/users").isValid("my-user","my-secret-password");

true

Removeusers.remove(user)

RemovesanexistingArangoDBuserfromthedatabase.

TheusernamemustbespecifiedinUserandthespecifiedusermustexistinthedatabase.

Thismethodwillfailiftheusercannotbefoundinthedatabase.

Examples

arangosh>require("@arangodb/users").remove("my-user");

Documentusers.document(user)

FetchesanexistingArangoDBuserfromthedatabase.

Theusernamemustbespecifiedinuser.


Examples

arangosh>require("@arangodb/users").document("my-user");


All

InArangosh

418

users.all()

FetchesallexistingArangoDBusersfromthedatabase.

Examples

arangosh>require("@arangodb/users").all();


Reloadusers.reload()

Reloadstheuserauthenticationdataontheserver

Alluserauthenticationdataisloadedbytheserveronceonstartuponlyandiscachedafterthat.Whenusersgetaddedordeleted,acacheflushisdoneautomatically,andthiscanbeperformedbyacalltothismethod.

Examples

arangosh>require("@arangodb/users").reload();

Permissionusers.permission(user,database[,collection])

Fetchestheaccessleveltothedatabaseoracollection.

Theuseranddatabasenamemustbespecified,optionallyyoucanspecifythecollectionname.


Examples

arangosh>require("@arangodb/users").permission("my-user","testdb");

rw

InArangosh

419

Command-lineoptions

GeneralOptions

Generalhelp--help

-h

Printsalistofthemostcommonoptionsavailableandthenexits.Inordertoseealloptionsuse--help-all.

Tomakeuseofthestartupoptionsfromaprogram,theoption--dump-optionswillprintoutalloptionsinJSONformatandthenexits.

Version

--version

-v

Printstheversionoftheserverandexits.

ConfigurationFiles

Optionscanbespecifiedonthecommandlineorinconfigurationfiles.IfastringVariableoccursinthevalue,itisreplacedbythecorrespondingenvironmentvariable.

--configurationfilename

-cfilename

Specifiesthenameoftheconfigurationfiletouse.

Ifthiscommandisnotpassedtotheserver,thenbydefault,theserverwillattempttofirstlocateafilenamed~/.arango/arangod.confintheuser'shomedirectory.

Ifnosuchfileisfound,theserverwillproceedtolookforafilearangod.confinthesystemconfigurationdirectory.Thesystemconfigurationdirectoryisplatform-specific,andmaybechangedwhencompilingArangoDByourself.Itmaydefaultto/etc/arangodbor/usr/local/etc/arangodb.Thisfileisinstalledwhenusingapackagemanagerlikerpmordpkg.IfyoumodifythisfileandlaterupgradetoanewversionofArangoDB,thenthepackagemanagernormallywarnsyouabouttheconflict.Inordertoavoidthesewarningforsmalladjustments,youcanputlocaloverridesintoafilearangod.conf.local.

Onlycommandlineoptionswithavalueshouldbesetwithintheconfigurationfile.Commandlineoptionswhichactasflagsshouldbeenteredonthecommandlinewhenstartingtheserver.

Eachoptionisspecifiedonaseparatelineintheform:

key=value

Alternatively,aheadersectioncanbespecifiedandoptionspertainingtothatsectioncanbespecifiedinashorterform

[log]

level=trace

ratherthanspecifying

log.level=trace

Soyouseeingeneral--section.paramvaluetranslatesto

[section]

param=value

ServerConfiguration

420

Whitespacearound=isignoredintheconfigurationfile.Donotputspacesaroundadditional=intheparametervaluehowever.Thefollowingexampleshowsthecorrectwaytospecifyalogleveloftraceforthetopicstartup:

log.level=startup=trace

Notethatthereisnowhitespacebetweenstartupand=,andalsonot=andtrace.

Whereonesectionmayoccurmultipletimes,andthelastoccuranceofparamwillbecomethefinalvalue.Incaseofparametersbeingvectors,multipleoccuranceaddsanotheritemtothevector.Vectorscanbeidentifiedbythe...inthe--helpoutputofthebinaries.

Commentscanbeplacedintheconfigurationfile,onlyifthelinebeginswithoneormorehashsymbols(#).

Theremaybeoccasionswhereaconfigurationfileexistsandtheuserwishestooverrideconfigurationsettingsstoredinaconfigurationfile.Anysettingsspecifiedonthecommandlinewilloverwritethesamesettingwhenitappearsinaconfigurationfile.Iftheuserwishestocompletelyignoreconfigurationfileswithoutnecessarilydeletingthefile(orfiles),thenaddthecommandlineoption

-cnone

or

--configurationnone

whenstartinguptheserver.Notethat,thewordnoneiscase-insensitive.

ServerConfiguration

421


FileSystems

(LINUX)

WerecommendtonotuseBTRFSonlinux,it'sknowntonotworkwellinconjunctionwithArangoDB.WeexperiencedthatarangodbfacinglatencyissuesonaccessingitsdatabasefilesonBTRFSpartitions.InconjunctionwithBTRFSandAUFSwealsosawdatalossonrestart.

VirtualMemoryPageSizes(LINUX)

Bydefault,ArangoDBusesJemallocasthememoryallocator.Jemallocdoesagoodjobofreducingvirtualmemoryfragmentation,especiallyforlong-runningprocesses.Unfortunately,someOSconfigurationscaninterferewithJemalloc'sabilitytofunctionproperly.Specifically,Linux's"transparenthugepages",Windows'"largepages"andothersimilarfeaturessometimespreventJemallocfromreturningunusedmemorytotheoperatingsystemandresultinunnecessarilyhighmemoryuse.Therefore,werecommenddisablingthesefeatureswhenusingJemallocwithArangoDB.Pleaseconsultyouroperatingsystem'sdocumentationforhowtodothis.

Execute

sudobash-c"echomadvise>/sys/kernel/mm/transparent_hugepage/enabled"

sudobash-c"echomadvise>/sys/kernel/mm/transparent_hugepage/defrag"

beforeexecutingarangod.

SwapSpace

(LINUX)

Itisrecommendedtoassignswapspaceforaserverthatisrunningarangod.Configuringswapspacecanpreventtheoperatingsystem'sOOMkillerfromkillingArangoDBtooeagerlyonLinux.

Over-CommitMemory

FortheMMFilesstorageengine,execute

sudobash-c"echo0>/proc/sys/vm/overcommit_memory"


FortheRocksDBstorageengine,execute

sudobash-c"echo2>/proc/sys/vm/overcommit_memory"

beforestarting.

Fromwww.kernel.org:

Whenthisflagis0,thekernelattemptstoestimatetheamountoffreememoryleftwhenuserspacerequestsmorememory.

Whenthisflagis1,thekernelpretendsthereisalwaysenoughmemoryuntilitactuallyrunsout.

Whenthisflagis2,thekernelusesa"neverovercommit"policythatattemptstopreventanyovercommitofmemory.


422

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Notethatthenusinganovercommit_memorysettingof2,thiswillbydefaultallowprocessestouseallswapspacebutonlyhalfoftheavailableRAM.Thiscanbechangedbyadjustingthevalueofovercommit_ratioaswell.

Fromwww.kernel.org:

Whenovercommit_memoryissetto2,thecommittedaddressspaceisnotpermittedtoexceedswapplusthispercentageofphysicalRAM.

ZoneReclaim

Execute

sudobash-c"echo0>/proc/sys/vm/zone_reclaim_mode"


Fromwww.kernel.org:

ThisisvalueORedtogetherof

1=Zonereclaimon2=Zonereclaimwritesdirtypagesout4=Zonereclaimswapspages

NUMAMulti-processorsystemsoftenhavenon-uniformAccessMemory(NUMA).ArangoDBshouldbestartedwithinterleaveonsuchsystem.Thiscanbeachievedusing

numactl--interleave=allarangod...

MaxMemoryMappings(LINUX)

Linuxkernelsbydefaultrestrictthemaximumnumberofmemorymappingsofasingleprocesstoabout64Kmappings.Whilethisvalueissufficientformostworkloads,itmaybetoolowforaprocessthathaslotsofparallelthreadsthatallrequiretheirownmemorymappings.Inthiscaseallthethreads'memorymappingswillbeaccountedtothesinglearangodprocess,andthemaximumnumberof64Kmappingsmaybereached.Whenthemaximumnumberofmappingsisreached,callstommapwillfail,sotheprocesswillthinknomorememoryisavailablealthoughtheremaybeplentyofRAMleft.

Toavoidthisscenario,itisrecommendedtoraisethedefaultvalueforthemaximumnumberofmemorymappingstoasufficientlyhighvalue.Asaruleofthumb,onecoulduse8timesthenumberofavailablecorestimes8,000.

Fora32coreserver,agoodrule-of-thumbvaluethuswouldbe2,048,000(3288000).Forcertainworkloads,itmaybesensibletouseevenahighervalueforthenumberofmemorymappings.

Tosetthevalueonce,usethefollowingcommandbeforestartingarangod:

sudobash-c"sysctl-w'vm.max_map_count=2048000'"

Tomakethesettingsdurable,itwillbenecessarytostoretheadjustedsettingsin/etc/sysctl.conforotherplacesthattheoperatingsystemislookingat.

EnvironmentVariables

(LINUX)


423



ItisrecommendedtosettheenvironmentvariableGLIBCXX_FORCE_NEWto1onsystemsthatuseglibc++inordertodisablethememorypoolingbuiltintoglibc++.ThatmemorypoolingisunnecessarybecauseJemallocwillalreadydomemorypooling.

Execute

exportGLIBCXX_FORCE_NEW=1

beforestartingarangod.

32bit

WhileitispossibletocompileArangoDBon32bitsystem,thisisnotarecommendedenvironment.64bitsystemscanaddressasignificantlybiggermemoryregion.


424

ManagingEndpointsTheArangoDBservercanlistenforincomingrequestsonmultipleendpoints.

TheendpointsarenormallyspecifiedeitherinArangoDB'sconfigurationfileoronthecommand-line,usingthe--server.endpoint.ArangoDBsupportsdifferenttypesofendpoints:

tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryptionunix:///path/to/socket-Unixdomainsocketendpoint

IfaTCP/IPendpointisspecifiedwithoutaportnumber,thenthedefaultport(8529)willbeused.Ifmultipleendpointsneedtobeused,theoptioncanberepeatedmultipletimes.

ThedefaultendpointforArangoDBistcp://127.0.0.1:8529ortcp://localhost:8529.

EXAMPLES

unix>./arangod--server.endpointtcp://127.0.0.1:8529

--server.endpointssl://127.0.0.1:8530

--ssl.keyfileserver.pem/tmp/vocbase

2012-07-26T07:07:47Z[8161]INFOusingSSLprotocolversion'TLSv1'

2012-07-26T07:07:48Z[8161]INFOusingendpoint'ssl://127.0.0.1:8530'forhttpsslrequests

2012-07-26T07:07:48Z[8161]INFOusingendpoint'tcp://127.0.0.1:8529'forhttptcprequests

2012-07-26T07:07:49Z[8161]INFOArangoDB(version1.1.alpha)isreadyforbusiness

2012-07-26T07:07:49Z[8161]INFOHaveFun!

TCPEndpoints

Givenahostname:

--server.endpointtcp://hostname:port

GivenanIPv4address:

--server.endpointtcp://ipv4-address:port

GivenanIPv6address:

--server.endpointtcp://[ipv6-address]:port

Ononespecificethernetinterfaceeachportcanonlybeboundexactlyonce.YoucanlookupyouravailableinterfacesusingtheifconfigcommandonLinux/MacOSX-theWindowsequivalentisipconfig(SeeWikipediaformoredetails).ThegeneralnamesoftheinterfacesdifferonOS'sandhardwarestheyrunon.However,typicallyeveryhosthasasocalledloopbackinterface,whichisavirtualinterface.Byconventionitalwayshastheaddress127.0.0.1or::1(ipv6),andcanonlybereachedfromexactlytheverysamehost.Ethernetinterfacesusuallyhavenameslikeeth0,wlan0,eth1:17,le0oraplaintextnameinWindows.

Tofindoutwhichservicesalreadyuseports(soArangoDBcan'tbindthemanymore),youcanusethenetstatcommand(itbehavesalittledifferentoneachplatform,runitwith-lnptonLinux,-ptcponMacOSXorwith-anonwindowsforvaluableinformation).

ArangoDBcanalsodoasocalledbroadcastbindusingtcp://0.0.0.0:8529.Thiswayitwillbereachableonallinterfacesofthehost.Thismaybeusefulondevelopmentsystemsthatfrequentlychangetheirnetworksetuplikelaptops.

SpecialnoteonIPv6link-localaddresses

ArangoDBcanalsolistentoIPv6link-localaddressesviaaddingthezoneIDtotheIPv6addressintheform[ipv6-link-local-address%zone-id].However,whatyouprobablyinsteadwantistobindtoalocalIPv6address.LocalIPv6addressesstartwithfd.Ifyouonlyseeafe80:IPv6addressinyourinterfaceconfigurationbutnoIPv6addressstartingwithfdyourinterfacehasnolocalIPv6addressassigned.YoucanreadmoreaboutIPv6link-localaddresseshere.

Example

ManagingEndpoints

425

http://en.wikipedia.org/wiki/Ifconfig

http://en.wikipedia.org/wiki/Loop_device

http://en.wikipedia.org/wiki/Netstat

https://en.wikipedia.org/wiki/Link-local_address#IPv6

Bindtoalink-localandlocalIPv6address.

unix>ifconfig

Thiscommandlistsallinterfacesandassignedipaddresses.Thelink-localaddressmaybefe80::6257:18ff:fe82:3ec6%eth0(IPv6addressplusinterfacename).AlocalIPv6addressmaybefd12:3456::789a.TobindArangoDBtoitstartarangodwith--server.endpointtcp://[fe80::6257:18ff:fe82:3ec6%eth0]:8529.Usetelnettotesttheconnection.

unix>telnetfe80::6257:18ff:fe82:3ec6%eth08529

Tryingfe80::6257:18ff:fe82:3ec6...

Connectedtomy-machine.

Escapecharacteris'^]'.

GET/HTTP/1.1

HTTP/1.1301MovedPermanently

Location:/_db/_system/_admin/aardvark/index.html

Content-Type:text/html

Server:ArangoDB

Connection:Keep-Alive

Content-Length:197

<html><head><title>Moved</title></head><body><h1>Moved</h1><p>Thispagehasmovedto<ahref="/_db/_system/_admin/aardvark/inde

x.html">/_db/_system/_admin/aardvark/index.html</a>.</p></body></html>

Reuseaddress--tcp.reuse-address

IfthisbooleanoptionissettotruethenthesocketoptionSO_REUSEADDRissetonallserverendpoints,whichisthedefault.Ifthisoptionissettofalseitispossiblethatittakesuptoaminuteafteraserverhasterminateduntilitispossibleforanewservertousethesameendpointagain.Thisiswhythisisactivatedbydefault.

Pleasenotehoweverthatundersomeoperatingsystemsthiscanbeasecurityriskbecauseitmightbepossibleforanotherprocesstobindtothesameaddressandport,possiblyhijackingnetworktraffic.UnderWindows,ArangoDBadditionallysetstheflagSO_EXCLUSIVEADDRUSEasameasuretoalleviatethisproblem.

Backlogsize--tcp.backlog-size

AllowstospecifythesizeofthebacklogforthelistensystemcallThedefaultvalueis10.Themaximumvalueisplatform-dependent.Specifyingahighervaluethandefinedinthesystemheader'sSOMAXCONNmayresultinawarningonserverstart.Theactualvalueusedbylistenmayalsobesilentlytruncatedonsomeplatforms(thishappensinsidethelistensystemcall).

ManagingEndpoints

426

SSLConfiguration

SSLEndpoints

Givenahostname:

--server.endpointtcp://hostname:port

GivenanIPv4address:

--server.endpointtcp://ipv4-address:port

GivenanIPv6address:

--server.endpointtcp://[ipv6-address]:port

Note:IfyouareusingSSL-encryptedendpoints,youmustalsosupplythepathtoaservercertificateusingthe--ssl.keyfileoption.

Keyfile--ssl.keyfilefilename

IfSSLencryptionisused,thisoptionmustbeusedtospecifythefilenameoftheserverprivatekey.ThefilemustbePEMformattedandcontainboththecertificateandtheserver'sprivatekey.

Thefilespecifiedbyfilenamecanbegeneratedusingopenssl:

#createprivatekeyinfile"server.key"

opensslgenrsa-des3-outserver.key1024

#createcertificatesigningrequest(csr)infile"server.csr"

opensslreq-new-keyserver.key-outserver.csr

#copyawayoriginalprivatekeyto"server.key.org"

cpserver.keyserver.key.org

#removepassphrasefromtheprivatekey

opensslrsa-inserver.key.org-outserver.key

#signthecsrwiththekey,createscertificatePEMfile"server.crt"

opensslx509-req-days365-inserver.csr-signkeyserver.key-outserver.crt

#combinecertificateandkeyintosinglePEMfile"server.pem"

catserver.crtserver.key>server.pem

YoumayusecertificatesissuedbyaCertificateAuthorityorself-signedcertificates.Self-signedcertificatescanbecreatedbyatoolofyourchoice.WhenusingOpenSSLforcreatingtheself-signedcertificate,thefollowingcommandsshouldcreateavalidkeyfile:

-----BEGINCERTIFICATE-----

(base64encodedcertificate)

-----ENDCERTIFICATE-----

-----BEGINRSAPRIVATEKEY-----

(base64encodedprivatekey)

-----ENDRSAPRIVATEKEY-----

Forfurtherinformationpleasecheckthemanualsofthetoolsyouusetocreatethecertificate.

CAFile--ssl.cafilefilename

SSLConfiguration

427

ThisoptioncanbeusedtospecifyafilewithCAcertificatesthataresenttotheclientwhenevertheserverrequestsaclientcertificate.Ifthefileisspecified,TheserverwillonlyacceptclientrequestswithcertificatesissuedbytheseCAs.Donotspecifythisoptionifyouwantclientstobeabletoconnectwithoutspecificcertificates.

ThecertificatesinfilenamemustbePEMformatted.

SSLprotocol

--ssl.protocolvalue

Usethisoptiontospecifythedefaultencryptionprotocoltobeused.Thefollowingvariantsareavailable:

1:SSLv22:SSLv2orSSLv3(negotiated)3:SSLv34:TLSv15:TLSv1.2

Thedefaultvalueis5(TLSv1.2).

SSLcache--ssl.session-cachevalue

SettotrueifSSLsessioncachingshouldbeused.

valuehasadefaultvalueoffalse(i.e.nocaching).

SSLpeercertificate

ThisfeatureisavailableintheEnterpriseEdition.

--ssl.require-peer-certificate

Requireapeercertificatefromtheclientbeforeconnecting.

SSLoptions

--ssl.optionsvalue

ThisoptioncanbeusedtosetvariousSSL-relatedoptions.IndividualoptionvaluesmustbecombinedusingbitwiseOR.

WhichoptionsareavailableonyourplatformisdeterminedbytheOpenSSLversionyouuse.Thelistofoptionsavailableonyourplatformmightberetrievedbythefollowingshellcommand:

>grep"#defineSSL_OP_.*"/usr/include/openssl/ssl.h

#defineSSL_OP_MICROSOFT_SESS_ID_BUG0x00000001L

#defineSSL_OP_NETSCAPE_CHALLENGE_BUG0x00000002L

#defineSSL_OP_LEGACY_SERVER_CONNECT0x00000004L

#defineSSL_OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG0x00000008L

#defineSSL_OP_SSLREF2_REUSE_CERT_TYPE_BUG0x00000010L

#defineSSL_OP_MICROSOFT_BIG_SSLV3_BUFFER0x00000020L

...

AdescriptionoftheoptionscanbefoundonlineintheOpenSSLdocumentation

SSLcipher--ssl.cipher-listcipher-list

ThisoptioncanbeusedtorestricttheservertocertainSSLciphersonly,andtodefinetherelativeusagepreferenceofSSLciphers.

Theformatofcipher-listisdocumentedintheOpenSSLdocumentation.

Tocheckwhichciphersareavailableonyourplatform,youmayusethefollowingshellcommand:

SSLConfiguration

428

http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html

>opensslciphers-v

ECDHE-RSA-AES256-SHASSLv3Kx=ECDHAu=RSAEnc=AES(256)Mac=SHA1

ECDHE-ECDSA-AES256-SHASSLv3Kx=ECDHAu=ECDSAEnc=AES(256)Mac=SHA1

DHE-RSA-AES256-SHASSLv3Kx=DHAu=RSAEnc=AES(256)Mac=SHA1

DHE-DSS-AES256-SHASSLv3Kx=DHAu=DSSEnc=AES(256)Mac=SHA1

DHE-RSA-CAMELLIA256-SHASSLv3Kx=DHAu=RSAEnc=Camellia(256)

Mac=SHA1

...

Thedefaultvalueforcipher-listis"ALL".

SSLConfiguration

429

LDAPThisfeatureisonlyavailableintheEnterpriseEdition.

BasicsConcepts

ThebasicideaisthatonecankeeptheuserauthenticationsetupforanArangoDBinstance(singleorcluster)outsideofArangoDBinanLDAPserver.AcrucialfeatureofthisisthatonecanaddandwithdrawusersandpermissionsbyonlychangingtheLDAPserverandinparticularwithouttouchingtheArangoDBinstance.ChangeswillbeeffectiveinArangoDBwithinafewminutes.

SincetherearemanydifferentpossibleLDAPsetups,wemustsupportavarietyofpossibilitiesforauthenticationandauthorization.Hereisashortoverview:

TomapArangoDBusernamestoLDAPuserstherearetwoauthenticationmethodscalled"simple"and"search".Inthe"simple"methodtheLDAPbinduserisderivedfromtheArangoDBusernamebyprependingaprefixandappendingasuffix.Forexample,auser"alice"couldbemappedtothedistinguishednameuid=alice,dc=arangodb,dc=comtoperformtheLDAPbindandauthentication.SeeSimpleauthenticationmethodbelowfordetailsandconfigurationoptions.

Inthe"search"methodtherearetwophases.InPhase1agenericread-onlyadminLDAPuseraccountisusedtobindtotheLDAPserverfirstandsearchforanLDAPusermatchingtheArangoDBusername.InPhase2,theactualauthenticationisthenperformedagainsttheLDAPuserthatwasfoundinphase1.Bothmethodsaresensibleandarerecommendedtouseinproduction.SeeSearchauthenticationmethodbelowfordetailsandconfigurationoptions.

Oncetheuserisauthenticated,therearenowtwomethodsforauthorization:(a)"rolesattribute"and(b)"rolessearch".

Inmethod(a)ArangoDBacquiresalistofrolestheauthenticatedLDAPuserhasfromtheLDAPserver.TheactualaccessrightstodatabasesandcollectionsfortheserolesareconfiguredinArangoDBitself.Theusereffectivelyhastheunionofallaccessrightsofallroleshehas.Thismethodisprobablythemostcommononeforproductionusecases.ItcombinestheadvantagesofmanagingusersandrolesoutsideofArangoDBintheLDAPserverwiththefinegrainedaccesscontrolwithinArangoDBfortheindividualroles.SeeRolesattributebelowfordetailsaboutmethod(a)andfortheassociatedconfigurationoptions.

Method(b)isverysimilarandonlydiffersfrom(a)inthewaytheactuallistofrolesofauserisderivedfromtheLDAPserver.SeeRolessearchbelowfordetailsaboutmethod(b)andfortheassociatedconfigurationoptions.

Fundamentaloptions

ThefundamentaloptionsforspecifyinghowtoaccesstheLDAPserverarethefollowing:

--ldap.enabledthisisabooleanoptionwhichmustbesettotruetoactivatetheLDAPfeature--ldap.serverisastringspecifyingthehostnameorIPaddressoftheLDAPserver--ldap.portisanintegerspecifyingtheporttheLDAPserverisrunningon,thedefaultis389--ldap.basednspecifiesthebasedistinguishednameunderwhichthesearchtakesplace(canalternativelybesetvia--ldap.url)--ldap.binddnand--ldap.bindpasswdaredistinguishednameandpasswordforaread-onlyLDAPusertowhichArangoDBcanbindtosearchtheLDAPserver.Notethatitisnecessarytoconfiguretheseforboththe"simple"and"search"authenticationmethods,sinceeveninthe"simple"method,ArangoDBoccasionallyhastorefreshtheauthorizationinformationfromtheLDAPservereveniftheusersessionpersistsandnonewauthenticationisneeded!Itis,however,allowedtoleavebothempty,butthentheLDAPservermustbereadablewithanonymousaccess.--ldap.refresh-rateisafloatingpointvalueinseconds.Thedefaultis300,whichmeansthatArangoDBwillrefreshtheauthorizationinformationforauthenticatedusersafteratmost5minutes.ThismeansthatchangesintheLDAPserverlikeremovedusersoraddedorremovedrolesforauserwillbeeffectiveafteratmost5minutes.

Notethatthe--ldap.serverand--ldap.portoptionscanalternativelybespecifiedinthe--ldap.urlstringtogetherwithotherconfigurationoptions.FordetailsseeSection"LDAPURLs"below.

HereisanexampleonhowtoconfiguretheconnectiontotheLDAPserver,withanonymousbind:

--ldap.enabled=true\

--ldap.server=ldap.arangodb.com\

LDAPOptions

430

--ldap.basedn=dc=arangodb,dc=com

WiththisconfigurationArangoDBbindsanonymouslytotheLDAPserveronhostldap.arangodb.comonthedefaultport389andexecutesallsearchesunderthebasedistinguishednamedc=arangodb,dc=com.

IfweneedausertoreadinLDAPhereistheexampleforit:



--ldap.basedn=dc=arangodb,dc=com\

--ldap.binddn=uid=arangoadmin,dc=arangodb,dc=com\

--ldap.bindpasswd=supersecretpassword

Theconnectionisidenticalbutthesearcheswillbeexecutedwiththegivendistinguishednameinbinddn.

Notehere:Thegivenuser(ortheanonymousone)needsatleastreadaccessonalluserobjectstofindthemandinthecaseofRolessearchalsoreadaccessontheobjectsstoringtheroles.

UptothispointArangoDBcannowconnecttoagivenLDAPserverbutitisnotyetabletoauthenticateusersproperlywithit.Forthispickoneofthefollowingtwoauthenticationmethods.

LDAPURLs

AsanalternativeonecanspecifythevaluesofmultipleLDAPrelatedconfigurationoptionsbyspecifyingasingleLDAPURL.Hereisanexample:

--ldap.urlldap://ldap.arangodb.com:1234/dc=arangodb,dc=com?uid?sub

Thisoneoptionhasthecombinedeffectofsettingthefollowing:


--ldap.port=1234\


--ldap.searchAttribute=uid\

--ldap.searchScope=sub

Thatis,theLDAPURLconsistsoftheLDAPserverandport,abasedn,asearchattributeandascopewhichcanbeoneofbase,oneorsub.Thereisalsothepossibilitytousetheldapsprotocolasin:

--ldap.urlldaps://ldap.arangodb.com:636/dc=arangodb,dc=com?uid?sub

Thisdoesexactlythesameastheoneabove,exceptthatitusestheLDAPoverTLSprotocol.Thisisanon-standardmethodwhichdoesnotinvolveusingtheSTARTTLSprotocol.NotethatthisdoesnotworkintheWindowsversion!WesuggesttousetheldapprotocolandSTARTTLSasdescribedinthenextsection.

TLSoptions

TLSisnotsupportedintheWindowsversionofArangoDB!

ToconfiguretheusageofencryptedTLStocommunicatewiththeLDAPserverthefollowingoptionsareavailable:

--ldap.tlsThemainswitchtoactiveTLS.caneitherbetrue(useTLS)orfalse(donotuseTLS).Itisswitchedoffbydefault.IfyouswitchthisonanddonotusetheldapsprotocolviatheLDAPURL,thenArangoDBwillusetheSTARTTLSprotocoltoinitiateTLS.Thisistherecommendedapproach.--ldap.tls-versiontheminimalTLSversionthatArangoDBshouldaccept.Availableversionsare1.0,1.1and1.2.Thedefaultis1.2.IfyourLDAPserverdoesnotsupportVersion1.2,youhavetochangethissetting.--ldap.tls-cert-check-strategystrategytovalidatetheLDAPservercertificate.Availablestrategiesarenever,hard,demand,allowandtry.Thedefaultishard.

LDAPOptions

431

--ldap.tls-cacert-fileAfilepathtooneormore(concatenated)certificateauthoritycertificatesinPEMformat.Asdefaultnofilepathisconfigured.Thiscertificateisusedtovalidatetheserverresponse.--ldap.tls-cacert-dirAdirectorypathtocertificateauthoritycertificatesinc_rehashformat.Asdefaultnodirectorypathisconfigured.

AssumingyouhavetheTLSCAcertfilethatisgiventotheserverat/path/to/certificate.pem,hereisanexampleonhowtoconfigureTLS:

--ldap.tlstrue\

--ldap.tls-cacert-file/path/to/certificate.pem

YoucanuseTLSwithanyofthefollowingauthenticationmechanisms.

Authenticationmethods

InordertoauthenticateusersinLDAPwehavetwooptionsavailable.Weneedtopickexactlyonethem.

Simpleauthenticationmethod

Thesimpleauthenticationmethodisusedifandonlyifboththe--ldap.prefixand--ldap.suffixconfigurationoptionsarespecifiedandarenon-empty.Inallothercasesthe"search"authenticationmethodisused.

Inthe"simple"methodtheLDAPbinduserisderivedfromtheArangoDBusernamebyprependingthevalueofthe--ldap.prefixconfigurationoptionandbyappendingthevalueofthe--ldap.suffixconfigurationoption.Forexample,anArangoDBuser"alice"wouldbemappedtothedistinguishednameuid=alice,dc=arangodb,dc=comtoperformtheLDAPbindandauthentication,if--ldap.prefixissettouid=and--ldap.suffixissetto,dc=arangodb,dc=com.

ArangoDBbindstotheLDAPserverandauthenticateswiththedistinguishednameandthepasswordprovidedbytheclient.IftheLDAPserversuccessfullyverifiesthepasswordthentheuserisauthenticated.

IfyouwanttousethismethodaddthefollowingexampletoyourArangoDBconfigurationtogetherwiththefundamentalconfiguration:

--ldap.prefixuid=\

--ldap.suffix,dc=arangodb,dc=com

ThismethodwillauthenticateanLDAPuserwiththedistinguishedname{PREFIX}{USERNAME}{SUFFIX},inthiscaseforthearangouseraliceitwillsearchfor:uid=alice,dc=arangodb,dc=com.Thisdistinguishednamewillbeusedas{{USER}}fortheroleslateron.

Searchauthenticationmethod

Thesearchauthenticationmethodisusedifatleastoneofthetwooptions--ldap.prefixand--ldap.suffixisemptyornotspecified.ArangoDBusestheLDAPusercredentialsgivenbythe--ldap.binddnand--ldap.bindpasswdtoperformasearchforLDAPusers.Inthiscase,thevaluesoftheoptions--ldap.basedn,--ldap.search-attribute,--ldap.search-filterand--ldap.search-scopeareusedinthefollowingway:

--ldap.search-scopeisanLDAPsearchscopewithpossiblevaluesbase(justsearchthebasedistinguishedname),sub(recursivesearchunderthebasedistinguishedname)orone(searchthebase'simmediatechildren)(default:sub)--ldap.search-filterisanLDAPfilterexpressionwhichlimitsthesetofLDAPusersbeingconsidered(default:objectClass=*whichmeansallobjects)--ldap.search-attributespecifiestheattributeintheuserobjectswhichisusedtomatchtheArangoDBusername(default:uid)

Hereisanexampleonhowtoconfigurethesearchmethod.AssumewehaveuserslikethefollowingstoredinLDAP:

dn:uid=alice,dc=arangodb,dc=com

uid:alice

objectClass:inetOrgPerson

objectClass:organizationalPerson

objectClass:top

objectClass:person

LDAPOptions

432

https://www.openssl.org/docs/man1.0.2/apps/c_rehash.html

WhereuidistheusernameusedinArangoDB,andweonlysearchforobjectsoftypepersonthenwecanaddthefollowingtoourfundamentalLDAPconfiguration:

--ldap.search-attribute=uid\

--ldap.search-filter=objectClass=person

Thiswillusethesubsearchscopebydefaultandwillfindallpersonobjectswheretheuidisequaltothegivenusername.Fromthesethednwillbeextractedandusedas{{USER}}intheroleslateron.

FetchingrolesforauserAfterauthentication,thenextstepistoderiveauthorizationinformationfromtheauthenticatedLDAPuser.Inordertofetchtherolesandtherebytheaccessrightsforauserweagainhavetwopossibleoptionsandneedtopickoneofthem.Wecancombineeachauthenticationmethodwitheachrolemethod.Inanycaseausercanhavenoroleormorethanone.IfauserhasnoroletheuserwillnotgetanyaccesstoArangoDBatall.Ifauserhasmultipleroleswithdifferentrightsthentherightswillbecombinedandthestrongestrightwillwin.Example:

alicehastherolesproject-aandproject-b.project-ahasnoaccesstocollectionBData.project-bhasrwaccesstocollectionBData,hencealicewillhaverwonBData.

NotethattheactualdatabaseandcollectionaccessrightswillbeconfiguredinArangoDBitselfbyrolesintheusersmodule.Therolenameisalwaysprefixedwith:role:,e.g.::role:project-aand:role:project-brespectively.YoucanusethenormaluserpermissionstoolsintheWebinterfaceorarangoshtoconfigurethese.

Rolesattribute

ThemostimportantmethodforthisistoreadofftherolesanLDAPuserisassociatedwithfromanattributeintheLDAPuserobject.Iftheconfigurationoption

--ldap.roles-attribute-name

configurationoptionisset,thenthevalueofthatoptionisthenameoftheattributebeingused.

Hereistheexampletoaddtotheoverallconfiguration:

--ldap.roles-attribute-name=role

IfwehavetheuserstoredlikethefollowinginLDAP:

dn:uid=alice,dc=arangodb,dc=com

uid:alice

objectClass:inetOrgPerson

objectClass:organizationalPerson

objectClass:top

objectClass:person

role:project-a

role:project-b

Thentherequestwillgranttherolesproject-aandproject-bfortheuseraliceaftersuccessfulauthentication,astheyarestoredwithintheroleontheuserobject.

Rolessearch

AnalternativemethodforauthorizationistoconductasearchintheLDAPserverforLDAPobjectsrepresentingrolesauserhas.Iftheconfigurationoption

--ldap.roles-search=<search-expression>

LDAPOptions

433

isgiven,thenthestring{USER}in<search-expression>isreplacedwiththedistinguishednameoftheauthenticatedLDAPuserandtheresultingsearchexpressionisusedtomatchdistinguishednamesofLDAPobjectsrepresentingrolesofthatuser.

Example:

--ldap.roles-search'(&(objectClass=groupOfUniqueNames)(uniqueMember={USER}))'

AfteraLDAPuserwasfoundandauthenticatedasdescribedintheauthenticationsectionabovethe{USER}inthesearchexpressionwillbereplacedbyitsdistinguishedname,e.g.uid=alice,dc=arangodb,dc=com,andthuswiththeabovesearchexpressiontheactualsearchexpressionwouldendupbeing:

(&(objectClass=groupOfUniqueNames)(uniqueMember=uid=alice,dc=arangodb,dc=com}))

ThissearchwillfindallobjectsofgroupOfUniqueNameswhereatleastoneuniqueMemberhasthednofalice.Thelistofresultsofthatsearchwouldbethelistofrolesgivenbythevaluesofthednattributesofthefoundroleobjects.

Roletransformationsandfilters

Forbothoftheaboveauthorizationmethodstherearefurtherconfigurationoptionstotunetherolelookup.Inthissectionwedescribethesefurtheroptions:

--ldap.roles-includecanbeusedtospecifyaregularexpressionthatisusedtofilterroles.Onlyrolesthatmatchtheregularexpressionareused.

--ldap.roles-excludecanbeusedtospecifyaregularexpressionthatisusedtofilterroles.Onlyrolesthatdonotmatchtheregularexpressionareused.

--ldap.roles-transformationcanbeusedtospecifyaregularexpressionandreplacementtextas/re/text/.Thisregularexpressionisappliedtotherolenamefound.Thisisespeciallyusefulintheroles-searchvarianttoextracttherealrolenameoutofthednvalue.

--ldap.superuser-rolecanbeusedtospecifytheroleassociatedwiththesuperuser.Anyuserbelongingtothisrolegainssuperuserstatus.Thisroleischeckedafterapplyingtheroles-transformationexpression.

Example:

--ldap.roles-include"^arangodb"

willonlyconsiderrolesthatstartwitharangodb.

--ldap.roles-exclude=disabled

willonlyconsiderrolesthatdocontaintheworddisabled.

--ldap.superuser-role"arangodb-admin"

anyonebelongingtothegroup"arangodb-admin"willbecomeasuperuser.

Theroles-transformationdeservesalargerexample.Assumeweareusingrolessearchandhavestoredrolesinthefollowingway:

dn:cn=project-a,dc=arangodb,dc=com

objectClass:top

objectClass:groupOfUniqueNames

uniqueMember:uid=alice,dc=arangodb,dc=com

uniqueMember:uid=bob,dc=arangodb,dc=com

cn:project-a

description:InternalprojectA

dn:cn=project-b,dc=arangodb,dc=com

objectClass:top

objectClass:groupOfUniqueNames

uniqueMember:uid=alice,dc=arangodb,dc=com

uniqueMember:uid=charlie,dc=arangodb,dc=com

LDAPOptions

434

cn:project-b

description:ExternalprojectB

Inthiscasewewillfindcn=project-a,dc=arangodb,dc=comasoneroleofalice.Howeverweactuallywanttoconfigurearolename::role:project-awhichiseasiertoreadandmaintainforouradministrators.

Ifwenowapplythefollowingtransformation:

--ldap.roles-transformation=/^cn=([^,]*),.*$/$1/

Theregexwillextractoutproject-aresp.project-bofthednattribute.

Incombinationwiththesuperuser-rolewecouldmakeallproject-amembersarangodbadminsbyusing:

--ldap.roles-transformation=/^cn=([^,]*),.*$/$1/\

--ldap.superuser-role=project-a

Completeconfigurationexamples

InthissectionwewouldliketopresentcompleteexamplesforasuccessfulLDAPconfigurationofArangoDB.Allofthefollowingarejustcombinationsofthedetailsdescribedabove.

Simpleauthenticationwithrole-search,usinganonymousLDAPuser

ThisexampleconnectstotheLDAPserverwithananonymousread-onlyuser.Weusethesimpleauthenticationmode(prefix+suffix)toauthenticateusersandapplyarolesearchforgroupOfUniqueNamesobjectswheretheuserisauniqueMember.Furthermoreweextractonlythecnoutofthedistinguishedrolename.




--ldap.prefixuid=\

--ldap.suffix,dc=arangodb,dc=com\

--ldap.roles-search'(&(objectClass=groupOfUniqueNames)(uniqueMember={USER}))'\

--ldap.roles-transformation=/^cn=([^,]*),.*$/$1/\

--ldap.superuser-role=project-a

SearchauthenticationwithrolesattributeusingLDAPadminuserhavingTLSenabled

ThisexampleconnectstotheLDAPserverwithagivendistinguishednameofanadminuser+password.FurthermoreweactivateTLSandgivethecertificatefiletovalidateserverresponses.Weusethesearchauthenticationsearchingfortheuidattributeofpersonobjects.Thesepersonobjectshaveroleattribute(s)containingtherole(s)ofauser.




--ldap.binddn=uid=arangoadmin,dc=arangodb,dc=com\

--ldap.bindpasswd=supersecretpassword\

--ldap.tlstrue\

--ldap.tls-cacert-file/path/to/certificate.pem\

--ldap.search-attribute=uid\

--ldap.search-filter=objectClass=person\

--ldap.roles-attribute-name=role

LDAPOptions

435

Command-LineOptionsforLogging

Loglevelsandtopics

ArangoDB'slogoutputisgroupedintotopics.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.Thelogverbosityandoutputfilescanbeadjustedperlogtopic.Forexample

--log.levelstartup=trace--log.levelqueries=trace--log.levelinfo

willlogmessagesconcerningstartupattracelevel,AQLqueriesattracelevelandeverythingelseatinfolevel.

Inaconfigurationfile,itiswrittenlikethis:

[log]

level=startup=trace

level=queries=trace

level=info

Notethattheremustnotbeanywhitespacearoundthesecond=.

Theavailableloglevelsare:

fatal:onlylogsfatalerrorserror:onlylogserrorswarning:onlylogswarningsanderrorsinfo:logsinformationmessages,warningsanderrorsdebug:logsdebugandinformationmessages,warningsanderrorstrace:logstrace,debugandinformationmessages,warningsanderrors

Notethatlevelsdebugandtracewillbeveryverbose.

SomerelevantlogtopicsavailableinArangoDB3are:

agency:informationabouttheagencycollector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperations(includingmsync)performance:performance-releatedmessagesqueries:executedAQLqueries,slowqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads

Logoutputs

Thelogoption--log.output<definition>allowsdirectingtheglobalorper-topiclogoutputtodifferentoutputs.Theoutputdefinition<definition>canbeoneof

-forstdin+forstderrsyslog://<syslog-facility>

syslog://<syslog-facility>/<application-name>

file://<relative-path>

Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputfordifferentlogtopics.Tosetupaper-topicoutputconfiguration,use--log.output<topic>=<definition>,e.g.

LoggingOptions

436

queries=file://queries.txt

logsallqueriestothefile"queries.txt".

Theoldoption--log.fileisstillavailablein3.0forconveniencereasons.In3.0itisashortcutforthemoregeneraloption--log.outputfile://filename.

Theoldoption--log.requests-fileisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.outputrequests=file://....

Using--log.outputalsoallowsdirectinglogoutputtodifferentfilesbasedontopics.Forexample,tologallAQLqueriestoafile"queries.log"onecanusetheoptions:

--log.levelqueries=trace--log.outputqueries=file:///path/to/queries.log

ToadditionallylogHTTPrequesttoafilenamed"requests.log"addtheoptions:

--log.levelrequests=info--log.outputrequests=file:///path/to/requests.log

Forcingdirectoutput

Theoption--log.force-directcanbeusedtodisablelogginginanextraloggingthread.Ifsettotrue,anylogmessagesareimmediatelyprintedinthethreadthattriggeredthelogmessage.Thisisnon-optimalforperformancebutcanaiddebugging.Ifsettofalse,logmessagesarehandedofftoanextraloggingthread,whichasynchronouslywritesthelogmessages.

Localtime

Logdatesandtimesinlocaltimezone:--log.use-local-time

Ifspecified,alldatesandtimesinlogmessageswillusetheserver'slocaltime-zone.Ifnotspecified,alldatesandtimesinlogmessageswillbeprintedinUTC/Zulutime.ThedateandtimeformatusedinlogsisalwaysYYYY-MM-DDHH:MM:SS,regardlessofthissetting.IfUTCtimeisused,aZwillbeappendedtoindicateZulutime.

Colorlogging--log.colorvalue

Loggingtoterminaloutputisbydefaultcolored.Colorfulloggingcanbeturnedoffbysettingthevaluetofalse.

SourcefileandLinenumber

Loglinenumber:--log.line-number

Normally,ifanhumanreadablefatal,error,warningorinfomessageislogged,noinformationaboutthefileandlinenumberisprovided.Thefileandlinenumberisonlyloggedfordebugandtracemessage.Thisoptioncanbeusetoalwayslogthesepiecesofinformation.

Prefix

Logprefix:--log.prefixprefix

Thisoptionisusedspecifyanprefixtologgedtext.

Threads

Logthreadidentifier:--log.threadtrue

Wheneverlogoutputisgenerated,theprocessIDiswrittenaspartoftheloginformation.Settingthisoptionappendsthethreadidofthecallingthreadtotheprocessid.Forexample,

2010-09-20T13:04:01Z[19355]INFOreadyforbusiness

LoggingOptions

437

whennothreadisloggedand

2010-09-20T13:04:17Z[19371-18446744072487317056]readyforbusiness

whenthiscommandlineoptionisset.

Toalsologthreadnames,itispossibletosetthe--log.thread-nameoption.Bydefault--log.thread-nameissettofalse.

Role

Logrole:--log.roletrue

Whensettotrue,thisoptionwillmaketheArangoDBloggerprintasinglecharacterwiththeserver'sroleintoeachloggedmessage.Therolesare:

U:undefined/unclear(usedatstartup)S:singleserverC:coordinatorP:primaryA:agent

Thedefaultvalueforthisoptionisfalse,sonoroleswillbelogged.

LoggingOptions

438

GeneralOptions

DatabaseUpgrade

--database.auto-upgrade

Specifyingthisoptionwillmaketheserverperformadatabaseupgradeinsteadofstartingtheservernormally.AdatabaseupgradewillfirstcomparetheversionnumberstoredinthefileVERSIONinthedatabasedirectorywiththecurrentserverversion.

Iftheversionnumberfoundinthedatabasedirectoryishigherthantheversionnumbertheserverisrunning,theserverexpectsthisisanunintentionaldowngradeandwillwarnaboutthis.Usingtheserverintheseconditionsisneitherrecommendednorsupported.

Iftheversionnumberfoundinthedatabasedirectoryislowerthantheversionnumbertheserverisrunning,theserverwillcheckwhetherthereareanyupgradetaskstoperform.Itwillthenexecuteallrequiredupgradetasksandprinttheirstatuses.Ifoneoftheupgradetasksfails,theserverwillexitwithanerror.Re-startingtheserverwiththeupgradeoptionwillthenagaintriggertheupgradecheckandexecutionuntiltheproblemisfixed.

Whetherornotthisoptionisspecified,theserverwillalwaysperformaversioncheckonstartup.Runningtheserverwithanon-matchingversionnumberintheVERSIONfilewillmaketheserverrefusetostart.

StorageEngine

AsofArangoDB3.2twostorageenginesaresupported.The"traditional"engineiscalledMMFiles,whichisalsothedefaultstorageengine.

AnalternativeenginebasedonRocksDBisalsoprovidedandcanbeturnedonmanually.

Onestorageenginetypeissupportedperserverperinstallation.Liveswitchingofstorageenginesonalreadyinstalledsystemsisn'tsupported.Configuringthewrongengine(notmatchingthepreviouslyusedone)willresultintheserverrefusingtostart.YoumayhoweveruseautotoletArangoDBchoosethepreviouslyusedone.

--server.storage-engine[auto|mmfiles|rocksdb]

Daemon

--daemon

Runstheserverasadaemon(asabackgroundprocess).Thisparametercanonlybesetifthepid(processid)fileisspecified.Thatis,unlessavaluetotheparameterpid-fileisgiven,thentheserverwillreportanerrorandexit.

DefaultLanguage--default-languagedefault-language

Thedefaultlanguageistusedforsortingandcomparingstrings.Thelanguagevalueisatwo-letterlanguagecode(ISO-639)oritiscomposedbyatwo-letterlanguagecodewithandatwolettercountrycode(ISO-3166).Validlanguagesare"de","en","en_US"or"en_UK".

Thedefaultdefault-languageissettobethesystemlocaleonthatplatform.

Supervisor

--supervisor

Executestheserverinsupervisormode.Intheeventthattheserverunexpectedlyterminatesduetoaninternalerror,thesupervisorwillautomaticallyrestarttheserver.Settingthisflagautomaticallyimpliesthattheserverwillrunasadaemon.Notethat,aswiththedaemonflag,thisflagrequiresthatthepid-fileparameterwillset.

unix>./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

2012-06-27T15:58:28Z[10133]INFOstartingupinsupervisormode

Ascanbeseen(e.g.byexecutingthepscommand),thiswillstartasupervisorprocessandtheactualdatabaseprocess:

GeneralOptions

439

http://rocksdb.org

unix>psfax|greparangod

10137?Ssl0:00./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

10142?Sl0:00\_./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

Whenthedatabaseprocessterminatesunexpectedly,thesupervisorprocesswillstartupanewdatabaseprocess:

>kill-SIGSEGV10142

>psfax|greparangod

10137?Ssl0:00./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

10168?Sl0:00\_./arangod--supervisor--pid-file/var/run/arangodb.pid/tmp/vocbase/

Useridentity

--uiduid

Thename(identity)oftheusertheserverwillrunas.Ifthisparameterisnotspecified,theserverwillnotattempttochangeitsUID,sothattheUIDusedbytheserverwillbethesameastheUIDoftheuserwhostartedtheserver.Ifthisparameterisspecified,thentheserverwillchangeitsUIDafteropeningportsandreadingconfigurationfiles,butbeforeacceptingconnectionsoropeningotherfiles(suchasrecoveryfiles).Thisisusefulwhentheservermustbestartedwithraisedprivileges(incertainenvironments)butsecurityconsiderationsrequirethattheseprivilegesbedroppedoncetheserverhasstartedwork.

Observethatthisparametercannotbeusedtobypassoperatingsystemsecurity.Ingeneral,thisparameter(anditscorrespondingrelativegid)canlowerprivilegesbutnotraisethem.

Groupidentity

--gidgid

Thename(identity)ofthegrouptheserverwillrunas.Ifthisparameterisnotspecified,thentheserverwillnotattempttochangeitsGID,sothattheGIDtheserverrunsaswillbetheprimarygroupoftheuserwhostartedtheserver.Ifthisparameterisspecified,thentheserverwillchangeitsGIDafteropeningportsandreadingconfigurationfiles,butbeforeacceptingconnectionsoropeningotherfiles(suchasrecoveryfiles).

Thisparameterisrelatedtotheparameteruid.

Processidentity

--pid-filefilename

ThenameoftheprocessIDfiletousewhenrunningtheserverasadaemon.Thisparametermustbespecifiedifeithertheflagdaemonorsupervisorisset.

Checkmaxmemorymappings

--server.check-max-memory-mappingscanbeusedonLinuxtomakearangodcheckthenumberofmemorymappingscurrentlyusedbytheprocess(asreportedin/proc//maps)andcompareitwiththemaximumnumberofallowedmappingsasdeterminedby/proc/sys/vm/max_map_count.Ifthecurrentnumberofmemorymappingsgetsnearthemaximumallowedvalue,arangodwilllogawarninganddisallowthecreationoffurtherV8contextstemporarilyuntilthecurrentnumberofmappingsgoesdownagain.

Iftheoptionissettofalse,nosuchcheckswillbeperformed.Allnon-Linuxoperatingsystemsdonotprovidethisoptionandwillignoreit.

Console--console

Runstheserverinanexclusiveemergencyconsolemode.Whenstartingtheserverwiththisoption,theserverisstartedwithaninteractiveJavaScriptemergencyconsole,withallnetworkingandHTTPinterfacesoftheserverdisabled.

Norequestscanbemadetotheserverinthismode,andtheonlywaytoworkwiththeserverinthismodeisbyusingtheemergencyconsole.Notethattheservercannotbestartedinthismodeifitisalreadyrunninginthisoranothermode.

GeneralOptions

440

RandomGenerator

--random.generatorarg

Theargumentisaninteger(1,2,3or4)whichsetsthemannerinwhichrandomnumbersaregenerated.Thedefaultmethod(3)istousetheanon-blockingrandom(orpseudorandom)numbergeneratorsuppliedbytheoperatingsystem.

Specifyinganargumentof2,usesablockingrandom(orpseudorandom)numbergenerator.Specifyinganargument1setsapseudorandomnumbergeneratorusinganimplicationoftheMersenneTwisterMT19937algorithm.Algorithm4isacombinationoftheblockingrandomnumbergeneratorandtheMersenneTwister.

Enable/disableauthentication

--server.authentication

Settingthisoptiontofalsewillturnoffauthenticationontheserversidesoallclientscanexecuteanyactionwithoutauthorizationandprivilegechecks.

Thedefaultvalueistrue.

JWTSecret

--server.jwt-secretsecret

ArangoDBwilluseJWTstoauthenticaterequests.UsingthisoptionletsyouspecifyaJWT.

InsingleserversetupsandwhennotspecifyingthissecretArangoDBwillgenerateasecret.

Inclusterdeploymentswhichhaveauthenticationenabledasecretmustbesetconsistentlyacrossallclustertaskssotheycantalktoeachother.

Enable/disableauthenticationforUNIXdomainsockets

--server.authentication-unix-socketsvalue

SettingvaluetotruewillturnoffauthenticationontheserversideforrequestscominginviaUNIXdomainsockets.Withthisflagenabled,clientslocatedonthesamehostastheArangoDBservercanuseUNIXdomainsocketstoconnecttotheserverwithoutauthentication.Requestscominginbyothermeans(e.g.TCP/IP)arenotaffectedbythisoption.

Thedefaultvalueisfalse.

Note:thisoptionisonlyavailableonplatformsthatsupportUNIXdomainsockets.

Enable/disableauthenticationforsystemAPIrequestsonly--server.authentication-system-onlyboolean

ControlswhetherincomingrequestsneedauthenticationonlyiftheyaredirectedtotheArangoDB'sinternalAPIsandfeatures,locatedat/_api/,/_admin/etc.

Iftheflagissettotrue,thenHTTPauthenticationisonlyrequiredforrequestsgoingtoURLsstartingwith/_,butnotforotherURLs.Theflagcanthusbeusedtoexposeauser-madeAPIwithoutHTTPauthenticationtotheoutsideworld,buttopreventtheoutsideworldfromusingtheArangoDBAPIandtheadmininterfacewithoutauthentication.NotethatcheckingtheURLisperformedafteranydatabasenameprefixhasbeenremoved.ThatmeanswhentheactualURLcalledis/_db/_system/myapp/myaction,theURL/myapp/myactionwillbeusedforauthentication-system-onlycheck.

Thedefaultistrue.

NotethatauthenticationstillneedstobeenabledfortheserverregularlyinorderforHTTPauthenticationtobeforcedfortheArangoDBAPIandthewebinterface.Settingonlythisflagisnotenough.

YoucancontrolArangoDB'sgeneralauthenticationfeaturewiththe--server.authenticationflag.

Enableauthenticationcachetimeout--server.authentication-timeoutvalue

GeneralOptions

441

Setsthecachetimeouttovalue(inseconds).ThisisonlynecessaryifyouuseanexternalauthenticationsystemlikeLDAP.

Enablelocalauthentication

--server.local-authenticationvalue

Ifsettofalseonlyusetheexternalauthenticationsystem.Iftruealsousethelocal_userscollections.

Thedefaultvalueistrue.

Enable/disablereplicationapplier--database.replication-applierflag

Iffalsetheserverwillstartwithreplicationappliersturnedoff,evenifthereplicationappliersareconfiguredwiththeautoStartoption.Usingthecommand-lineoptionwillnotchangethevalueoftheautoStartoptionintheapplierconfiguration,butwillsuppressauto-startingthereplicationapplierjustonce.

Iftheoptionisnotused,ArangoDBwillreadtheapplierconfigurationfromthefileREPLICATION-APPLIER-CONFIGonstartup,andusethevalueoftheautoStartattributefromthisfile.

Thedefaultistrue.

Keep-alivetimeout

--http.keep-alive-timeout

AllowstospecifythetimeoutforHTTPkeep-aliveconnections.Thetimeoutvaluemustbespecifiedinseconds.Idlekeep-aliveconnectionswillbeclosedbytheserverautomaticallywhenthetimeoutisreached.Akeep-alive-timeoutvalue0willdisablethekeepalivefeatureentirely.

HideProductheader

--http.hide-product-header

Iftrue,theserverwillexcludetheHTTPheader"Server:ArangoDB"inHTTPresponses.Ifsettofalse,theserverwillsendtheheaderinresponses.

Thedefaultisfalse.

Allowmethodoverride

--http.allow-method-override

Whenthisoptionissettotrue,theHTTPrequestmethodwilloptionallybefetchedfromoneofthefollowingHTTPrequestheadersifpresentintherequest:

x-http-methodx-http-method-overridex-method-override

Iftheoptionissettotrueandanyoftheseheadersisset,therequestmethodwillbeoverriddenbythevalueoftheheader.Forexample,thisallowsissuinganHTTPDELETErequestwhichtotheoutsideworldwilllooklikeanHTTPGETrequest.Thisallowsbypassingproxiesandtoolsthatwillonlyletcertainrequesttypespass.

Settingthisoptiontotruemayimposeasecurityrisksoitshouldonlybeusedincontrolledenvironments.

Thedefaultvalueforthisoptionisfalse.

Serverthreads

--server.threadsnumber

Specifiesthenumberofthreadsthatarespawnedtohandlerequests.

GeneralOptions

442

Togglingserverstatistics

--server.statisticsvalue

Ifthisoptionisvalueisfalse,thenArangoDB'sstatisticsgatheringisturnedoff.StatisticsgatheringcausesregularCPUactivitysousingthisoptiontoturnitoffmightrelieveheavy-loadedinstancesabit.

Sessiontimeout

timetoliveforserversessions--server.session-timeoutvalue

Thetimeoutforwebinterfacesessions,usingforauthenticatingrequeststothewebinterface(/_admin/aardvark)andrelatedareas.

Sessionsareonlyusedwhenauthenticationisturnedon.

Foxxqueues

enableordisabletheFoxxqueuesfeature--foxx.queuesflag

Iftrue,theFoxxqueueswillbeavailableandjobsinthequeueswillbeexecutedasynchronously.

Thedefaultistrue.Whensettofalsethequeuemanagerwillbedisabledandanyjobsarepreventedfrombeingprocessed,whichmayreduceCPUloadabit.

Foxxqueuespollinterval

pollintervalforFoxxqueues--foxx.queues-poll-intervalvalue

ThepollintervalfortheFoxxqueuesmanager.Thevalueisspecifiedinseconds.LowervalueswillmeanmoreimmediateandmorefrequentFoxxqueuejobexecution,butwillmakethequeuethreadwakeupandquerythequeuesmoreoften.Whensettoalowvalue,thequeuethreadmightcauseCPUload.

Thedefaultis1second.IfFoxxqueuesarenotusedmuch,thenthisvaluemaybeincreasedtomakethequeuesthreadwakeupless.

Directory

--database.directorydirectory

Thedirectorycontainingthecollectionsanddatafiles.Defaultsto/var/lib/arango.Whenspecifyingthedatabasedirectory,pleasemakesurethedirectoryisactuallywritablebythearangodprocess.

YoushouldfurthernotuseadatabasedirectorywhichisprovidedbyanetworkfilesystemsuchasNFS.Thereasonisthatnetworkedfilesystemsmightcauseinconsistencieswhentherearemultipleparallelreadersorwritersortheylackfeaturesrequiredbyarangod(e.g.flock()).

directory

Whenusingthecommandlineversion,youcansimplysupplythedatabasedirectoryasargument.

Examples

>./arangod--server.endpointtcp://127.0.0.1:8529--database.directory

/tmp/vocbase

Databasedirectorystateprecondition

--database.require-directory-statestate

Usingthisoptionitispossibletorequirethedatabasedirectorytobeinaspecificstateonstartup.theoptionsforthisvalueare:

non-existing:databasedirectorymustnotexistexisting:databasedirectorymustexistempty:databasedirectorymustexistbutbeemptypopulated:databasedirectorymustexistandcontainspecificfilesalreadyany:anydirectorystateallowed

GeneralOptions

443

Journalsize

--database.maximal-journal-sizesize

Maximalsizeofjournalinbytes.Canbeoverwrittenwhencreatinganewcollection.Notethatthisalsolimitsthemaximalsizeofasingledocument.

Thedefaultis32MB.

Waitforsync

defaultwaitforsyncbehavior--database.wait-for-syncboolean

Defaultwait-for-syncvalue.Canbeoverwrittenwhencreatinganewcollection.

Thedefaultisfalse.

Forcesyncingofproperties

forcesyncingofcollectionpropertiestodisk--database.force-sync-propertiesboolean

Forcesyncingofcollectionpropertiestodiskaftercreatingacollectionorupdatingitsproperties.

Ifturnedoff,nofsyncwillhappenforthecollectionanddatabasepropertiesstoredinparameter.jsonfilesinthefilesystem.Turningoffthisoptionwillspeedupworkloadsthatcreateanddropalotofcollections(e.g.testsuites).

Thedefaultistrue.

LimitingmemoryforAQLqueries

--query.memory-limitvalue

Thedefaultmaximumamountofmemory(inbytes)thatasingleAQLquerycanuse.WhenasingleAQLqueryreachesthespecifiedlimitvalue,thequerywillbeabortedwitharesourcelimitexceededexception.Inacluster,thememoryaccountingisdonepershard,sothelimitvalueiseffectivelyamemorylimitperquerypershard.

ThegloballimitvaluecanbeoverridenperquerybysettingthememoryLimitoptionvalueforindividualquerieswhenrunninganAQLquery.

Thedefaultvalueis0,meaningthatthereisnomemorylimit.

TurningAQLwarningsintoerrors--query.fail-on-warningvalue

Whensettotrue,AQLqueriesthatproducewarningswillinstantlyabortandthrowanexception.ThisoptioncanbesettocatchobviousissueswithAQLqueriesearly.Whensettofalse,AQLqueriesthatproducewarningswillnotabortandreturnthewarningsalongwiththequeryresults.TheoptioncanalsobeoverriddenforeachindividualAQLquery.

Enable/disableAQLquerytracking

--query.trackingflag

Iftrue,theserver'sAQLslowquerytrackingfeaturewillbeenabledbydefault.Trackingofqueriescanbedisabledbysettingtheoptiontofalse.

Thedefaultistrue.

Enable/disabletrackingofbindvariablesinAQLqueries--query.tracking-with-bindvarsflag

Iftrue,thenthebindvariableswillbetrackedforallrunningandslowAQLqueries.Thisoptiononlyhasaneffectif--query.trackingwassettotrue.Trackingofbindvariablescanbedisabledbysettingtheoptiontofalse.

Thedefaultistrue.

GeneralOptions

444

ThresholdforslowAQLqueries

--query.slow-thresholdvalue

BysettingvalueitcanbecontrolledafterwhatexecutiontimeanAQLqueryisconsidered"slow".Anyslowqueriesthatexceedtheexecutiontimespecifiedinvaluewillbeloggedwhentheyarefinished.Thethresholdvalueisspecifiedinseconds.Trackingofslowqueriescanbeturnedoffentirelybysettingtheoption--query.trackingtofalse.

Thedefaultvalueis10.0.

Queryregistrytimeout

--query.registry-ttlvalue

ThedefaulttimeoutforAQLquerypartstostayaliveinthecluster.Thedefaultvalueis600seconds.Querypartsthatarenotusedfortheconfiguredamountoftimewillexpireautomaticallyandwillbeaborted.Thevalueofthisoptionnormallyonlyneedstobeincreasedforqueriesthatarerunninglongerthanthedefaulttimeoutvalue(600seconds)andthattimeout.Theoptionhasnoeffectinsingle-servermode.

Throwcollectionnotloadederror--database.throw-collection-not-loaded-errorflag

Accessinganot-yetloadedcollectionwillautomaticallyloadacollectiononfirstaccess.Thisflagcontrolswhathappensincaseanoperationwouldneedtowaitforanotherthreadtofinalizeloadingacollection.Ifsettotrue,thenthefirstoperationthataccessesanunloadedcollectionwillloadit.Furtherthreadsthattrytoaccessthesamecollectionwhileitisstillloadingwillgetanerror(1238,collectionnotloaded).Whentheinitialoperationhascompletedloadingthecollection,alloperationsonthecollectioncanbecarriedoutnormally,anderror1238willnotbethrown.

Ifsettofalse,thefirstthreadthataccessesanot-yetloadedcollectionwillstillloadit.Otherthreadsthattrytoaccessthecollectionwhileloadingwillnotfailwitherror1238butinsteadblockuntilthecollectionisfullyloaded.Thisconfigurationmightleadtoallserverthreadsbeingblockedbecausetheyareallwaitingforthesamecollectiontocompleteloading.Settingtheoptiontotruewillpreventthisfromhappening,butrequiresclientstocatcherror1238andreactonit(maybebyschedulingaretryforlater).

Thedefaultvalueisfalse.

AQLQuerycachingmode--query.cache-mode

TogglestheAQLquerycachebehavior.Possiblevaluesare:

off:donotusequerycacheon:alwaysusequerycache,exceptforqueriesthathavetheircacheattributesettofalsedemand:usequerycacheonlyforqueriesthathavetheircacheattributesettotrue

AQLQuerycachesize--query.cache-entries

Maximumnumberofqueryresultsthatcanbestoredperdatabase-specificquerycache.Ifaqueryiseligibleforcachingandthenumberofitemsinthedatabase'squerycacheisequaltothisthresholdvalue,anothercachedqueryresultwillberemovedfromthecache.

Thisoptiononlyhasaneffectifthequerycachemodeissettoeitheronordemand.

JavaScriptcodeexecution--javascript.allow-admin-execute

Thisoptioncanbeusedtocontrolwhetheruser-definedJavaScriptcodeisallowedtobeexecutedonserverbysendingviaHTTPtotheAPIendpoint/_admin/executewithanauthenticateduseraccount.Thedefaultvalueisfalse,whichdisablestheexecutionofuser-definedcode.Thisisalsotherecommendedsettingforproduction.Intestenvironments,itmaybeconvenienttoturntheoptiononinordertosendarbitrarysetuporteardowncommandsforexecutionontheserver.

GeneralOptions

445

V8contexts

--javascript.v8-contextsnumber

SpecifiesthemaximumnumberofV8contextsthatarecreatedforexecutingJavaScriptcode.MorecontextsallowexecutingmoreJavaScriptactionsinparallel,providedthattherearealsoenoughthreadsavailable.PleasenotethateachV8contextwilluseasubstantialamountofmemoryandrequiresperiodicCPUprocessingtimeforgarbagecollection.

NotethatthisvalueconfiguresthemaximumnumberofV8contextsthatcanbeusedinparallel.UponserverstartonlyasmanyV8contextswillbecreatedasareconfiguredinoption--javascript.v8-contexts-minimum.TheactualnumberofavailableV8contextsmayfloatatruntimebetween--javascript.v8-contexts-minimumand--javascript.v8-contexts.WhenthereareunusedV8contextsthatlingeraround,theserver'sgarbagecollectorthreadwillautomaticallydeletethem.

--javascript.v8-contexts-minimumnumber

SpecifiestheminimumnumberofV8contextsthatwillbepresentatanytimetheserverisrunning.TheactualnumberofV8contextswillneverdropbelowthisvalue,butitmaygoupashighasspecifiedviatheoption--javascript.v8-contexts.

WhenthereareunusedV8contextsthatlingeraroundandthenumberofV8contextsisgreaterthan--javascript.v8-contexts-minimumtheserver'sgarbagecollectorthreadwillautomaticallydeletethem.

--javascript.v8-contexts-max-invocations

SpecifiesthemaximumnumberofinvocationsafterwhichausedV8contextisdisposed.Thedefaultvalueof--javascript.v8-contexts-max-invocationsis0,meaningthatthemaximumnumberofinvocationspercontextisunlimited.

--javascript.v8-contexts-max-age

Specifiesthetimeduration(inseconds)afterwhichtimeaV8contextisdisposedautomaticallyafteritscreation.Ifthetimeiselapsed,thecontextwillbedisposed.Thedefaultvaluefor--javascript.v8-contexts-max-ageis60seconds.

Ifboth--javascript.v8-contexts-max-invocationsand--javascript.v8-contexts-max-ageareset,thenthecontextwillbedestroyedwheneitherofthespecifiedthresholdvaluesisreached.

Garbagecollectionfrequency(time-based)

--javascript.gc-frequencyfrequency

Specifiesthefrequency(inseconds)fortheautomaticgarbagecollectionofJavaScriptobjects.Thissettingisusefultohavethegarbagecollectionstillworkinperiodswithnoorlittlenumbersofrequests.

Garbagecollectioninterval(request-based)--javascript.gc-intervalinterval

Specifiestheinterval(approximatelyinnumberofrequests)thatthegarbagecollectionforJavaScriptobjectswillberunineachthread.

V8options

--javascript.v8-optionsoptions

OptionalargumentstopasstotheV8Javascriptengine.TheV8enginewillrunwithdefaultsettingsunlessexplicitoptionsarespecifiedusingthisoption.TheoptionspassedwillbeforwardedtotheV8enginewhichwillparsethemonitsown.Passinginvalidoptionsmayresultinanerrorbeingprintedonstderrandtheoptionbeingignored.

Optionsneedtobepassedinonestring,withV8optionnamesbeingprefixedwithdoubledashes.Multipleoptionsneedtobeseparatedbywhitespace.TogetalistofallavailableV8options,youcanusethevalue"--help" asfollows:

--javascript.v8-options="--help"

AnotherexampleofspecificV8optionsbeingsetatstartup:

--javascript.v8-options="--log"

GeneralOptions

446

NamesandfeaturesorusableoptionsdependontheversionofV8beingused,andmightchangeinthefutureifadifferentversionofV8isbeingusedinArangoDB.NotalloptionsofferedbyV8mightbesensibletouseinthecontextofArangoDB.Usethespecificoptionsonlyifyouaresurethattheyarenotharmfulfortheregulardatabaseoperation.

GeneralOptions

447

MMFilesWrite-aheadlogoptionsSinceArangoDB2.2,theMMFilesstorageenginewillwritealldata-modificationoperationsintoitswrite-aheadlog.

WithArangoDB3.2anotherStorageengineoptionbecomesavailable-RocksDB.IncaseofusingRocksDBmostofthesubsequentoptionsdon'thaveausefulmeaning.

Thewrite-aheadlogisasequenceoflogfilesthatarewritteninanappend-onlyfashion.Fulllogfileswilleventuallybegarbage-collected,andtherelevantdatamightbetransferredintocollectionjournalsanddatafiles.Unneededandalreadygarbage-collectedlogfileswilleitherbedeletedorkeptforthepurposeofkeepingareplicationbacklog.

Directory

TheWALlogfilesdirectory:--wal.directory

Specifiesthedirectoryinwhichthewrite-aheadlogfilesshouldbestored.Ifthisoptionisnotspecified,itdefaultstothesubdirectoryjournalsintheserver'sglobaldatabasedirectory.Ifthedirectoryisnotpresent,itwillbecreated.

Logfilesize

thesizeofeachWALlogfile--wal.logfile-size

Specifiesthefilesize(inbytes)foreachwrite-aheadlogfile.Thelogfilesizeshouldbechosensothateachlogfilecanstoreaconsiderableamountofdocuments.Thebiggerthelogfilesizeischosen,thelongeritwilltaketofillupasinglelogfile,whichalsoinfluencesthedelayuntilthedatainalogfilewillbegarbage-collectedandwrittentocollectionjournalsanddatafiles.Italsoaffectshowlonglogfilerecoverywilltakeatserverstart.

Allowoversizeentries

whetherornotoversizeentriesareallowed--wal.allow-oversize-entries

Whetherornotitisallowedtostoreindividualdocumentsthatarebiggerthanwouldfitintoasinglelogfile.Settingtheoptiontofalsewillmakesuchoperationsfailwithanerror.Settingtheoptiontotruewillmakesuchoperationssucceed,butwithahighpotentialperformanceimpact.Thereasonisthatforeachoversizeoperation,anindividualoversizelogfileneedstobecreatedwhichmayalsoblockotheroperations.Theoptionshouldbesettofalseifitiscertainthatdocumentswillalwayshaveasizesmallerthanasinglelogfile.

Numberofreservelogfiles

maximumnumberofreservelogfiles--wal.reserve-logfiles

ThemaximumnumberofreservelogfilesthatArangoDBwillcreateinabackgroundprocess.Reservelogfilesareusefulinthesituationwhenanoperationneedstobewrittentoalogfilebutthereservespaceinthelogfileistoolowforstoringtheoperation.Inthiscase,anewlogfileneedstobecreatedtostoretheoperation.Creatingnewlogfilesisnormallyslow,soArangoDBwilltrytopre-createlogfilesinabackgroundprocesssotherearealwaysreservelogfileswhentheactivelogfilegetsfull.ThenumberofreservelogfilesthatArangoDBkeepsinthebackgroundisconfigurablewiththisoption.

Numberofhistoriclogfiles

maximumnumberofhistoriclogfiles--wal.historic-logfiles

ThemaximumnumberofhistoriclogfilesthatArangoDBwillkeepaftertheyhavebeengarbage-collected.Ifnoreplicationisused,thereisnoneedtokeephistoriclogfilesexceptforhavingalocalchangelog.

Inareplicationsetup,thenumberofhistoriclogfilesaffectstheamountofdataaslavecanfetchfromthemaster'slogs.Themorehistoriclogfiles,themorehistoricdataisavailableforaslave,whichisusefuliftheconnectionbetweenmasterandslaveisunstableorslow.Nothavingenoughhistoriclogfilesavailablemightleadtologfiledatabeingdeletedonthemasteralreadybeforeaslavehasfetchedit.

Syncinterval


448

intervalforautomatic,non-requesteddisksyncs--wal.sync-interval

Theinterval(inmilliseconds)thatArangoDBwillusetoautomaticallysynchronizedatainitswrite-aheadlogstodisk.Automaticsyncswillonlybeperformedfornot-yetsynchronizeddata,andonlyforoperationsthathavebeenexecutedwithoutthewaitForSyncattribute.

Flushtimeout

WALflushtimeout`--wal.flush-timeout

Thetimeout(inmilliseconds)thatArangoDBwillatmostwaitwhenflushingafullWALlogfiletodisk.Whenthetimeoutisreachedandtheflushisnotcompleted,theoperationthatrequestedtheflushwillfailwithalocktimeouterror.

Throttling

ThrottlewritestoWALwhenatleastsuchmanyoperationsarewaitingforgarbagecollection:--wal.throttle-when-pending

Themaximumvalueforthenumberofwrite-aheadloggarbage-collectionqueueelements.Ifsetto0,thequeuesizeisunbounded,andnowrite-throttlingwilloccur.Ifsettoanon-zerovalue,write-throttlingwillautomaticallykickinwhenthegarbage-collectionqueuecontainsatleastasmanyelementsasspecifiedbythisoption.Whilewrite-throttlingisactive,data-modificationoperationswillintentionallybedelayedbyaconfigurableamountoftime.Thisistoensurethewrite-aheadloggarbagecollectorcancatchupwiththeoperationsexecuted.Write-throttlingwillstayactiveuntilthegarbage-collectionqueuesizegoesdownbelowthespecifiedvalue.Write-throttlingisturnedoffbydefault.

--wal.throttle-wait

Thisoptiondeterminesthemaximumwaittime(inmilliseconds)foroperationsthatarewrite-throttled.Ifwrite-throttlingisactiveandanewwriteoperationistobeexecuted,itwillwaitforatmostthespecifiedamountoftimeforthewrite-aheadloggarbage-collectionqueuesizetofallbelowthethrottlingthreshold.Ifthequeuesizedecreasesbeforethemaximumwaittimeisover,theoperationwillbeexecutednormally.Ifthequeuesizedoesnotdecreasebeforethewaittimeisover,theoperationwillbeabortedwithanerror.Thisoptiononlyhasaneffectif--wal.throttle-when-pendinghasanon-zerovalue,whichisnotthedefault.

Numberofslots

Maximumnumberofslotstobeusedinparallel:--wal.slots

Configurestheamountofwriteslotsthewrite-aheadlogcangivetowriteoperationsinparallel.Anywriteoperationwillleaseaslotandreturnittothewrite-aheadlogwhenitisfinishedwritingthedata.Aslotwillremainblockeduntilthedatainitwassynchronizedtodisk.Afterthat,aslotbecomesreusablebyfollowingoperations.Therequirednumberofslotsisthusdeterminedbytheparallelityofwriteoperationsandthedisksynchronizationspeed.Slowdisksprobablyneedhighervalues,andfastdisksmayonlyrequireavaluelowerthanthedefault.

Ignorelogfileerrors

Ignorelogfileerrorswhenopeninglogfiles:--wal.ignore-logfile-errors

Ignoresanyrecoveryerrorscausedbycorruptedlogfilesonstartup.Whensettofalse,therecoveryprocedureonstartupwillfailwithanerrorwheneveritencountersacorrupted(thatincludesonlyhalf-written)logfile.Thisisasecurityprecautiontopreventdatalossincaseofdiskerrorsetc.Whentherecoveryprocedureabortsbecauseofcorruption,anycorruptedfilescanbeinspectedandfixed(orremoved)manuallyandtheservercanberestartedafterwards.

Settingtheoptiontotruewillmaketheservercontinuewiththerecoveryprocedureevenincaseitdetectscorruptlogfileentries.Inthiscaseitwillstopatthefirstcorruptedlogfileentryandignoreallothers,whichmightcausedataloss.

Ignorerecoveryerrors

Ignorerecoveryerrors:--wal.ignore-recovery-errors

Ignoresanyrecoveryerrorsnotcausedbycorruptedlogfilesbutbylogicalerrors.Logicalerrorscanoccuriflogfilesoranyotherserverdatafileshavebeenmanuallyeditedortheserverissomehowmisconfigured.


449

Ignore(non-WAL)datafileerrors

Ignoredatafileerrorswhenloadingcollections:--database.ignore-datafile-errorsboolean

Ifsettofalse,CRCmismatchandothererrorsincollectiondatafileswillleadtoacollectionnotbeingloadedatall.Thecollectioninthiscasebecomesunavailable.IfsuchcollectionneedstobeloadedduringWALrecovery,theWALrecoverywillalsoabort(ifnotforcedwithoption--wal.ignore-recovery-errorstrue).

Settingthisflagtofalseprotectsusersfromunintentionallyusingacollectionwithcorrupteddatafiles,fromwhichonlyasubsetoftheoriginaldatacanberecovered.Workingwithsuchcollectioncouldleadtodatalossandfollowuperrors.Inordertoaccesssuchcollection,itisrequiredtoinspectandrepairthecollectiondatafilewiththedatafiledebugger(arango-dfdb).

Ifsettotrue,CRCmismatchandothererrorsduringtheloadingofacollectionwillleadtothedatafilebeingpartiallyloaded,uptothepositionofthefirsterror.Alldatauptountiltheinvalidpositionwillbeloaded.Thiswillenableuserstocontinuewithcollectiondatafileseveniftheyarecorrupted,butthiswillresultinonlyapartialloadoftheoriginaldataandpotentialfollowuperrors.TheWALrecoverywillstillabortwhenencounteringacollectionwithacorrupteddatafile,atleastif--wal.ignore-recovery-errorsisnotsettotrue.

SettingtheoptiontotruewillalsoautomaticallrepairpotentiallycorruptedVERSIONfilesofdatabasesonstartup,sothatthestartupcanproceed.

Thedefaultvalueisfalse,socollectionswithcorrupteddatafileswillnotbeloadedatall,preventingpartialloadsandfollowuperrors.However,ifsuchcollectionisrequiredatserverstartup,duringWALrecovery,theserverwillaborttherecoveryandrefusetostart.


450

MMFilesCompactionoptionsTheArangoDBMMFilesstorageenginewillrunacompactionoverdatafiles.

ArangoDBwritesDocumentsintheWALfile.Oncetheyhavebeensealedinthewalfile,thecollectormaycopythemintoapercollectionjournalfile.

Oncejournalfilesfillup,they'resealedtobecomedatafiles.

=>onecollectionmayhavedocumentsintheWALlogs,itsjournalfile,andanarbitrarynumberofdatafiles.

Ifacollectionisloaded,eachofthesefilesareopened(thususeafilehandle)andaremmap'ed.Sincefilehandlesandmemorymappedfilesarealsoasparseresource,thatnumbershouldbekeptlow.

Onceyouupdateorremovedocumentsfromdatafiles(oralreadydidwhileitwasthejournalfile)thesedocumentsaremarkedas'dead'withadeletionmarker.

Overtimethenumberofdeaddocumentsmayrise,andwedon'twanttousethepreviouslymentionedresources,plusthediskspaceshouldbegivenbacktothesystem.Thusseveraljournalfilescanbecombinedtoone,ommittingthedeaddocuments.

Combiningseveralofthesedatafilesintooneiscalledcompaction.Thecompactionprocessreadsthealivedocumentsfromtheoriginaldatafiles,andwritesthemintonewdatafile.

Oncethatisdone,thememorymappingstotheolddatafilesisreleased,andthefilesareerased.

Sincethecompactionlocksthecollection,andalsousesI/Oresources,itscarefullyconfigurableunderwhichconditionsthesystemshouldperformwhichamountofthesecompactionjobs:

ArangoDBspawnsonecompactorthreadperdatabase.Thesettingsbelowvaryinscope.

ActivitycontrolTheactivitycontrolparametersalterthebehaviourintermsofscan/executionfrequencyofthecompaction.

Sleepintervalbetweentwocompactionruns(inseconds):--compaction.db-sleep-timeThenumberofsecondsthecollectorthreadwillwaitbetweentwoattemptstosearchforcompactabledatafilesofcollectionsinoneDatabase.Ifthecompactorhasactuallyexecutedwork,asubsequentlookupisdone.Scope:Database.

Minimumsleeptimebetweentwocompactionruns(inseconds):--compaction.min-intervalWhenanactualcompactionwasexecutedforonecollection,wewaitforthistimebeforeweexecutethecompactiononthiscollectionagain.Thisisheretoleteventuallypiledupuserloadbeworkedout.Scope:collection.

SourcedatafilesTheseparameterscontrolwhichdatafilesaretakenintoaccountforacompactionrun.Youcanspecifyseveralcriteriawhicheachoffmaybesufficcientalone.

Thescanoverthedatafilesbelongingtoonecollectionisexecutedfromoldestdatafiletonewest;iffilesqualifyforacompactiontheymaybemergedwithnewerfiles(containingyoungerdocuments)

Scope:Collectionlevel,someareinfluencedbycollectionsettings.

minimalfilesizethresholdoriginaldatafileshavetobebelowforacompaction:--compaction.min-small-data-file-sizeThisisthethresholdwhichcontrolsbelowwhichminimumtotalsizeadatafilewillalwaysbetakenintoaccountforthecompaction.

Minimumunusedcountofdocumentsinadatafile:--compaction.dead-documents-thresholdDatafileswilloftencontaindeaddocuments.Thisparameterspecifiestheirtopmostaccetpeablecountuntilthedatafilequalifiesforcompaction.

Howmanybytesofthesourcedatafileareallowedtobeunusedatmost:--compaction.dead-size-thresholdThedeaddatasizevariesalongwiththesizeofyourdocuments.Ifyouhavemanybigdocuments,thisthresholdmayhitbeforethedocumentcountthreshold.

CompactionOptions

451

Howmanypercentofthesourcedatafileshouldbeunusedatleast:--compaction.dead-size-percent-thresholdsincethesizeofthedocumentsmayvarythisthresholdworksonthepercentageofthedeaddocumentssize.Thus,ifyouhavemanyhugedeaddocuments,thisthresholdkicksinearlier.

Tonameanexamplewithnumbers,ifthedatafilecontains800kbytesofaliveand400kbytesofdeaddocuments,theshareofthedeaddocumentsis:

400/(400+800)=33%.

Ifthisvalueifhigherthanthespecifiedthreshold,thedatafilewillbecompacted.

CompactedtargetfilesOncedatafilesofacollectionarequalifiedforacompactionrun,theseparameterscontrolhowmanydatafilesaremergedintoone,(orevenonesourcedatafilemaybecompactedintoonesmallertargetdatafile)

Scope:Collectionlevel,someareinfluencedbycollectionsettings.

Maximumnumberoffilestomergetoonefile:--compaction.dest-max-filesHowmanydatafiles(atmost)wemaymergeintooneresultingdatafileduringonecompactionrun.

Howlargetheresultingfilemaybeincomparisontothecollectionsdatabase.maximal-journal-sizesetting:--compaction.dest-max-file-size-factorInArangoDByoucanconfigureadefaultjournalfilesizegloballyandoverrideitonapercollectionlevel.Thisvaluecontrolsthesizeofcollecteddatafilesrelativetotheconfiguredjournalfilesizeofthecollectioninquestion.

Afactorof3meansthatthemaximumfilesizeofthecompactedfileis3timesthesizeofthemaximumcollectionjournalfilesize.

howlargemaythecompactionresultfilebecome:--compaction.dest-max-result-file-sizenexttothefactorabove,atotallymaximumallowedfilesizeinbytesmaybespecified.Thiswilloverruleallpreviousparameters.

CompactionOptions

452

ClustersOptions

Agencyendpoint

Listofagencyendpoints:--cluster.agency-endpointendpoint

Anagencyendpointtheservercanconnectto.Theoptioncanbespecifiedmultipletimes,sotheservercanuseaclusterofagencyservers.Endpointshavethefollowingpattern:

tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryption

AtleastoneendpointmustbespecifiedorArangoDBwillrefusetostart.ItisrecommendedtospecifyatleasttwoendpointssoArangoDBhasanalternativeendpointifoneofthembecomesunavailable.

Examples

--cluster.agency-endpointtcp://192.168.1.1:4001--cluster.agency-endpointtcp://192.168.1.2:4002...

Myaddress

Thisserver'saddress/endpoint:--cluster.my-addressendpoint

Theserver'sendpointforcluster-internalcommunication.Ifspecified,itmusthavethefollowingpattern:

tcp://ipv4-address:port-TCP/IPendpoint,usingIPv4tcp://[ipv6-address]:port-TCP/IPendpoint,usingIPv6ssl://ipv4-address:port-TCP/IPendpoint,usingIPv4,SSLencryptionssl://[ipv6-address]:port-TCP/IPendpoint,usingIPv6,SSLencryption

Ifnoendpointisspecified,theserverwilllookupitsinternalendpointaddressintheagency.Ifnoendpointcanbefoundintheagencyfortheserver'sid,ArangoDBwillrefusetostart.

Examples

Listenonlyoninterfacewithaddress192.168.1.1

--cluster.my-addresstcp://192.168.1.1:8530

Listenonallipv4andipv6addresses,whichareconfiguredonport8530

--cluster.my-addressssl://[::]:8530

Myrole

Thisserver'srole:--cluster.my-role[dbserver|coordinator]

Theserver'srole.Isthisinstanceadbserver(backenddataserver)oracoordinator(frontendserverforexternalandapplicationaccess)

NodeID(deprecated)

Thisserver'sid:--cluster.my-local-infoinfo

Somelocalinformationabouttheserverinthecluster,thiscanforexamplebeanIPaddresswithaprocessIDoranystringuniquetotheserver.Specifyinginfoismandatoryonstartupiftheserverid(seebelow)isnotspecified.Eachserveroftheclustermusthaveauniquelocalinfo.Thisisignoredifmy-idbelowisspecified.

ClusterOptions

453

Thisoptionisdeprecatedandwillberemovedinafuturerelease.TheclusternodeidshavebeendroppedinfavourofoncegeneratedUUIDs.

Moreadvancedoptions(shouldgenerallyremainuntouched)

Synchroneousreplicationtiming:--cluster.synchronous-replication-timeout-factordouble

Strechorclinchtimeoutsforinternalsynchroneousreplicationmechanismbetweendbservers.Allsuchtimeoutsareaffectedbythischange.Pleasechangeonlywithintentandgreatcare.Defaultat1.0.

Systemreplicationfactor:--cluster.system-replication-factorinteger

Changedefaultreplicationfactorforsystemcollections.Defaultat2.

ClusterOptions

454

RocksDBengineoptionsRocksDBisahighlyconfigurablekey-valuestoreusedtopowerourRocksDBstorageengine.Mostoftheoptionsonthispagearepass-throughoptionstotheunderlyingRocksDBinstance,andwechangeveryfewoftheirdefaultsettings.

Dependingonthestorageengineyouhavechosentheavailabilityandthescopeoftheseoptionschanges.

Incaseyouhavechosenmmfilessomeofthefollowingoptionsapplytopersistentindexes.Incaseofrocksdbitwillapplytoalldatastoredaswellasindexes.

Pass-throughoptions--rocksdb.wal-directory

AbsolutepathfortheRocksDBWALfiles.Ifleftempty,thiswilluseasubdirectoryjournalsinsidethedatadirectory.

Writebuffers

--rocksdb.write-buffer-size

Theamountofdatatobuildupineachin-memorybuffer(backedbyalogfile)beforeclosingthebufferandqueuingittobeflushedintostandardstorage.Default:64MiB.Largervaluesmayimproveperformance,especiallyforbulkloads.

--rocksdb.max-write-buffer-number

Themaximumnumberofwritebuffersthatbuiltupinmemory.Ifthisnumberisreachedbeforethebufferscanbeflushed,writeswillbeslowedorstalled.Default:2.

--rocksdb.min-write-buffer-number-to-merge

Minimumnumberofwritebuffersthatwillbemergedtogetherwhenflushingtonormalstorage.Default:1.

--rocksdb.max-total-wal-size

MaximumtotalsizeofWALfilesthat,whenreached,willforceaflushofallcolumnfamilieswhosedataisbackedbytheoldestWALfiles.Settingthistoalowvaluewilltriggerregularflushingofcolumnfamilydatafrommemtables,sothatWALfilescanbemovedtothearchive.SettingthistoahighvaluewillavoidregularflushingbutmaypreventWALfilesfrombeingmovedtothearchiveandbeingremoved.

--rocksdb.delayed-write-rate(Hidden)

LimitedwriteratetoDB(inbytespersecond)ifwearewritingtothelastin-memorybufferallowedandweallowmorethan3buffers.Default:16MiB/s.

LSMtreestructure

--rocksdb.num-levels

ThenumberoflevelsforthedatabaseintheLSMtree.Default:7.

--rocksdb.num-uncompressed-levels

Thenumberoflevelsthatdonotusecompression.Thedefaultvalueis2.LevelsabovethisnumberwilluseSnappycompressiontoreducethediskspacerequirementsforstoringdataintheselevels.

--rocksdb.dynamic-level-bytes

Iftrue,theamountofdataineachleveloftheLSMtreeisdetermineddynamicallysoastominimizethespaceamplification;otherwise,thelevelsizesarefixed.ThedynamicsizingallowsRocksDBtomaintainawell-structuredLSMtreeregardlessoftotaldatasize.Default:true.

--rocksdb.max-bytes-for-level-base

Themaximumtotaldatasizeinbytesinlevel-1oftheLSMtree.Onlyeffectiveif--rocksdb.dynamic-level-bytesisfalse.Default:256MiB.


455

--rocksdb.max-bytes-for-level-multiplier

ThemaximumtotaldatasizeinbytesforlevelLoftheLSMtreecanbecalculatedasmax-bytes-for-level-base*(max-bytes-for-level-multiplier^(L-1)).Onlyeffectiveif--rocksdb.dynamic-level-bytesisfalse.Default:10.

--rocksdb.level0-compaction-trigger

Compactionoflevel-0tolevel-1istriggeredwhenthismanyfilesexistinlevel-0.Settingthistoahighernumbermayhelpbulkwritesattheexpenseofslowingdownreads.Default:2.

--rocksdb.level0-slowdown-trigger

Whenthismanyfilesaccumulateinlevel-0,writeswillbesloweddownto--rocksdb.delayed-write-ratetoallowcompactiontocatchup.Default:20.

--rocksdb.level0-stop-trigger

Whenthismanyfilesaccumulateinlevel-0,writeswillbestoppedtoallowcompactiontocatchup.Default:36.

FileI/O

--rocksdb.compaction-read-ahead-size

Ifnon-zero,weperformbiggerreadswhendoingcompaction.Ifyou'rerunningRocksDBonspinningdisks,youshouldsetthistoatleast2MiB.ThatwayRocksDB'scompactionisdoingsequentialinsteadofrandomreads.Default:0.

--rocksdb.use-direct-reads(Hidden)

OnlymeaningfulonLinux.Ifset,useO_DIRECTforreadingfiles.Default:false.

--rocksdb.use-direct-io-for-flush-and-compaction(Hidden)

OnlymeaningfulonLinux.Ifset,useO_DIRECTforwritingfiles.Default:false.

--rocksdb.use-fsync(Hidden)

Ifset,issueanfsynccallwhenwritingtodisk(settofalsetoissuefdatasynconly.Default:false.

Backgroundtasks

--rocksdb.max-background-jobs

Maximumnumberofconcurrentbackgroundcompactionjobs,submittedtothelowprioritythreadpool.Default:numberofprocessors.

--rocksdb.num-threads-priority-high

Numberofthreadsforhighpriorityoperations(e.g.flush).Werecommendsettingthisequaltomax-background-flushes.Default:numberofprocessors/2.

--rocksdb.num-threads-priority-low

Numberofthreadsforlowpriorityoperations(e.g.compaction).Default:numberofprocessors/2.

Caching

--rocksdb.block-cache-size

Thisisthesizeoftheblockcacheinbytes.Increasingthismayimproveperformance.Ifthereislessthan4GiBofRAMonthesystem,thedefaultvalueis256MiB.Ifthereismore,thedefaultis(systemRAMsize-2GiB)*0.3.

--rocksdb.block-cache-shard-bits

Thenumberofbitsusedtoshardtheblockcachetoallowconcurrentoperations.Tokeepindividualshardsatareasonablesize(i.e.atleast512KB),keepthisvaluetoatmostblock-cache-shard-bits/512KB.Default:block-cache-size/2^19.

--rocksdb.table-block-size

Approximatesizeofuserdata(inbytes)packedperblockforuncompresseddata.

--rocksdb.recycle-log-file-num(Hidden)

Numberoflogfilestokeeparoundforrecycling.Default:0.


456

Miscellaneous

--rocksdb.optimize-filters-for-hits(Hidden)

Thisflagspecifiesthattheimplementationshouldoptimizethefiltersmainlyforcaseswherekeysarefoundratherthanalsooptimizeforthecasewherekeysarenot.Thiswouldbeusedincaseswheretheapplicationknowsthatthereareveryfewmissesortheperformanceinthecaseofmissesisnotasimportant.Default:false.

--rocksdb.wal-recovery-skip-corrupted(Hidden)

Iftrue,skipcorruptedrecordsinWALrecovery.Default:false.

Non-Pass-ThroughOptions--rocksdb.wal-file-timeout(Hidden)

TimeoutafterwhichunusedWALfilesaredeleted(inseconds).Default:10.0s.

DataofongoingtransactionsisstoredinRAM.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatacreatedormodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.

ThefollowingoptionscanbeusedtocontroltheRAMusageandautomaticintermediatecommitsfortheRocksDBengine:

--rocksdb.max-transaction-size

Transactionsizelimit(inbytes).TransactionsstoreallkeysandvaluesinRAM,solargetransactionsruntheriskofcausingout-of-memorysitations.Thissettingallowsyoutoensurethatdoesnothappenbylimitingthesizeofanyindividualtransaction.TransactionswhoseoperationswouldconsumemoreRAMthanthisthresholdvaluewillabortautomaticallywitherror32("resourcelimitexceeded").

--rocksdb.intermediate-commit-size

Ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.

--rocksdb.intermediate-commit-count

Ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.

--rocksdb.throttle

Ifenabled,throttlestheingestrateofwritesifnecessarytoreducechancesofcompactionsgettingtoofarbehindandblockingincomingwrites.Thisoptionistruebydefault.


457

HashcacheoptionsSinceArangoDB3.2,theseveralcorecomponentsoftheserveruseacachesystemwhichpoolsmemoryacrossmanydifferentcachetables.Inordertoprovideintelligentinternalmemorymanagement,thesystemperiodicallyreclaimsmemoryfromcacheswhichareusedlessoftenandreallocatesittocacheswhichgetmoreactivity.

Cachesize

Globalsizelimitforallhashcaches:--cache.size

Theglobalcachingsystem,allcaches,andallthedatacontainedthereinwillfitinsidethislimit.Thesizeisspecifiedinbytes.Ifthereislessthan4GiBofRAMonthesystem,thedefaultvalueis256MiB.Ifthereismore,thedefaultis(systemRAMsize-2GiB)*0.3.

Rebalancinginterval

Timebetweencacherebalancingattempts:--cache.rebalancing-interval

Thevalueisspecifiedinmicrosecondswithadefaultof2secondsandaminimumof500milliseconds.

HashCacheOptions

458

AsynchronousTasks

maximalqueuesize

Maximumsizeofthequeueforrequests:--server.maximal-queue-sizesize

Specifiesthemaximumsizeofthequeueforasynchronoustaskexecution.Ifthequeuealreadycontainssizetasks,newtaskswillberejecteduntilothertasksarepoppedfromthequeue.Settingthisvaluemayhelppreventingfromrunningoutofmemoryifthequeueisfilledupfasterthantheservercanprocessrequests.

AsynchronousTasks

459

DurabilityConfiguration

GlobalConfiguration

Thereareglobalconfigurationvaluesfordurability,whichcanbeadjustedbyspecifyingthefollowingconfigurationoptions:

defaultwaitforsyncbehavior--database.wait-for-syncboolean

Defaultwait-for-syncvalue.Canbeoverwrittenwhencreatinganewcollection.

Thedefaultisfalse.

forcesyncingofcollectionpropertiestodisk--database.force-sync-propertiesboolean

Forcesyncingofcollectionpropertiestodiskaftercreatingacollectionorupdatingitsproperties.

Ifturnedoff,nofsyncwillhappenforthecollectionanddatabasepropertiesstoredinparameter.jsonfilesinthefilesystem.Turningoffthisoptionwillspeedupworkloadsthatcreateanddropalotofcollections(e.g.testsuites).

Thedefaultistrue.

intervalforautomatic,non-requesteddisksyncs--wal.sync-interval

Theinterval(inmilliseconds)thatArangoDBwillusetoautomaticallysynchronizedatainitswrite-aheadlogstodisk.Automaticsyncswillonlybeperformedfornot-yetsynchronizeddata,andonlyforoperationsthathavebeenexecutedwithoutthewaitForSyncattribute.

Per-collectionconfigurationYoucanalsoconfigurethedurabilitybehavioronaper-collectionbasis.UsetheArangoDBshelltochangetheseproperties.

getsorsetsthepropertiesofacollectioncollection.properties()

Returnsanobjectcontainingallcollectionproperties.



isVolatile:IftruethenthecollectiondatawillbekeptinmemoryonlyandArangoDBwillnotwriteorsyncthedatatodisk.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

keyOptions(optional)additionaloptionsforkeygeneration.ThisisaJSONarraycontainingthefollowingattributes(note:someoftheattributesareoptional):

type:thetypeofthekeygeneratorusedforthecollection.allowUserKeys:ifsettotrue,thenitisallowedtosupplyownkeyvaluesinthe_keyattributeofadocument.Ifsettofalse,thenthekeygeneratorwillsolelyberesponsibleforgeneratingkeysandsupplyingownkeyvaluesinthe_keyattributeofdocumentsisconsideredanerror.increment:incrementvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.offset:initialoffsetvalueforautoincrementkeygenerator.Notusedforotherkeygeneratortypes.

indexBuckets:numberofbucketsintowhichindexesusingahashtablearesplit.Thedefaultis16andthisnumberhastobeapowerof2andlessthanorequalto1024.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

Forverylargecollectionsoneshouldincreasethistoavoidlongpauseswhenthehashtablehastobeinitiallybuiltorresized,sincebucketsareresizedindividuallyandcanbeinitiallybuiltinparallel.Forexample,64mightbeasensiblevalueforacollectionwith100000000documents.Currently,onlytheedgeindexrespectsthisvalue,butotherindextypesmightfollowinfutureArangoDBversions.Changes(seebelow)areappliedwhenthecollectionisloadedthenexttime.

Inaclustersetup,theresultwillalsocontainthefollowingattributes:

numberOfShards:thenumberofshardsofthecollection.

Durability

460

shardKeys:containsthenamesofdocumentattributesthatareusedtodeterminethetargetshardfordocuments.

replicationFactor:determineshowmanycopiesofeachshardarekeptondifferentDBServers.

collection.properties(properties)

Changesthecollectionproperties.propertiesmustbeanobjectwithoneormoreofthefollowingattribute(s):



indexBuckets:Seeabove,changesareonlyappliedwhenthecollectionisloadedthenexttime.ThisoptionismeaningfulfortheMMFilesstorageengineonly.

replicationFactor:ChangethenumberofshardcopieskeptondifferentDBServers,validvaluesareintegernumbersintherangeof1-10(Clusteronly)

Note:itisnotpossibletochangethejournalsizeafterthejournalordatafilehasbeencreated.Changingthisparameterwillonlyeffectnewlycreatedjournals.Alsonotethatyoucannotlowerthejournalsizetolessthensizeofthelargestdocumentalreadystoredinthecollection.

Note:someothercollectionproperties,suchastype,isVolatile,orkeyOptionscannotbechangedoncethecollectioniscreated.

Examples

Readallproperties

arangosh>db.example.properties();

showexecutionresultsChangeaproperty

arangosh>db.example.properties({waitForSync:true});


Per-operationconfigurationManydata-modificationoperationsandalsoArangoDB'stransactionsallowtospecifyawaitForSyncattribute,whichwhensetensurestheoperationdatahasbeensynchronizedtodiskwhentheoperationreturns.

Disk-UsageConfigurationTheamountofdiskspaceusedbyArangoDBisdeterminedbyafewconfigurationoptions.

GlobalConfigurationThetotalamountofdiskstoragerequiredbyArangoDBisdeterminedbythesizeofthewrite-aheadlogfilesplusthesizesofthecollectionjournalsanddatafiles.

Therearethefollowingoptionsforconfiguringthenumberandsizesofthewrite-aheadlogfiles:

maximumnumberofreservelogfiles--wal.reserve-logfiles

ThemaximumnumberofreservelogfilesthatArangoDBwillcreateinabackgroundprocess.Reservelogfilesareusefulinthesituationwhenanoperationneedstobewrittentoalogfilebutthereservespaceinthelogfileistoolowforstoringtheoperation.Inthiscase,anewlogfileneedstobecreatedtostoretheoperation.Creatingnewlogfilesisnormallyslow,soArangoDBwilltrytopre-createlogfilesinabackgroundprocesssotherearealwaysreservelogfileswhentheactivelogfilegetsfull.ThenumberofreservelogfilesthatArangoDBkeepsinthebackgroundisconfigurablewiththisoption.

Durability

461

maximumnumberofhistoriclogfiles--wal.historic-logfiles

ThemaximumnumberofhistoriclogfilesthatArangoDBwillkeepaftertheyhavebeengarbage-collected.Ifnoreplicationisused,thereisnoneedtokeephistoriclogfilesexceptforhavingalocalchangelog.

Inareplicationsetup,thenumberofhistoriclogfilesaffectstheamountofdataaslavecanfetchfromthemaster'slogs.Themorehistoriclogfiles,themorehistoricdataisavailableforaslave,whichisusefuliftheconnectionbetweenmasterandslaveisunstableorslow.Nothavingenoughhistoriclogfilesavailablemightleadtologfiledatabeingdeletedonthemasteralreadybeforeaslavehasfetchedit.

thesizeofeachWALlogfile--wal.logfile-size

Specifiesthefilesize(inbytes)foreachwrite-aheadlogfile.Thelogfilesizeshouldbechosensothateachlogfilecanstoreaconsiderableamountofdocuments.Thebiggerthelogfilesizeischosen,thelongeritwilltaketofillupasinglelogfile,whichalsoinfluencesthedelayuntilthedatainalogfilewillbegarbage-collectedandwrittentocollectionjournalsanddatafiles.Italsoaffectshowlonglogfilerecoverywilltakeatserverstart.

whetherornotoversizeentriesareallowed--wal.allow-oversize-entries

Whetherornotitisallowedtostoreindividualdocumentsthatarebiggerthanwouldfitintoasinglelogfile.Settingtheoptiontofalsewillmakesuchoperationsfailwithanerror.Settingtheoptiontotruewillmakesuchoperationssucceed,butwithahighpotentialperformanceimpact.Thereasonisthatforeachoversizeoperation,anindividualoversizelogfileneedstobecreatedwhichmayalsoblockotheroperations.Theoptionshouldbesettofalseifitiscertainthatdocumentswillalwayshaveasizesmallerthanasinglelogfile.Whendatagetscopiedfromthewrite-aheadlogfilesintothejournalsordatafilesofcollections,fileswillbecreatedonthecollectionlevel.Howbigthesefilesareisdeterminedbythefollowingglobalconfigurationvalue:

--database.maximal-journal-sizesize

Maximalsizeofjournalinbytes.Canbeoverwrittenwhencreatinganewcollection.Notethatthisalsolimitsthemaximalsizeofasingledocument.

Thedefaultis32MB.

Per-collectionconfigurationThejournalsizecanalsobeadjustedonaper-collectionlevelusingthecollection'spropertiesmethod.

Durability

462

EncryptionThisfeatureisonlyavailableintheEnterpriseEdition.

WhenyoustoresensitivedatainyourArangoDBdatabase,youwanttoprotectthatdataunderallcircumstances.AtruntimeyouwillprotectitwithSSLtransportencryptionandstrongauthentication,butwhenthedataisalreadyondisk,youalsoneedprotection.ThatiswheretheEncryptionfeaturecomesin.

TheEncryptionfeatureofArangoDBwillencryptalldatathatArangoDBisstoringinyourdatabasebeforeitiswrittentodisk.

ThedataisencryptedwithAES-256-CTR,whichisastrongencryptionalgorithm,thatisverysuitableformulti-processorenvironments.Thismeansthatyourdataissafe,butyourdatabaseisstillfast,evenunderload.

MostmodernCPU'shavebuiltinsupportforhardwareAESencryption,whichmakesitevenfaster.

Note:TheEncryptionfeaturerequirestheRocksDBstorageengine.

Encryptionkeys

TheEncryptionfeatureofArangoDBrequiresasingle32-bytekeyperserver.Itisrecommendedtouseadifferentkeyforeachserver(whenoperatinginaclusterconfiguration).Makesuretoprotectthesekeys!

Thatmeans:

Donotwritethemtopersistentdisksoryourserver(s),alwaysstorethemonanin-memory(tmpfs)filesystem.Transportyourkeyssafelytoyourserver(s).Therearevarioustoolsformanagingsecretslikethis(e.g.vaultproject.io).Storeacopyofyourkeyofflineinasafeplace.Ifyouloseyourkey,thereisNOwaytogetyourdataback.

ConfigurationToactivateencryptionofyourdatabase,youneedtosupplyanencryptionkeytotheserver.

Makesuretopassthisoptiontheveryfirsttimeyoustartyourdatabase.Youcannotencryptadatabasethatalreadyexists.

Note:Youalsohavetoactivatetherocksdbstorageengine.

Encryptionkeystoredinfile

Passthefollowingoptiontoarangod:

$arangod\

--rocksdb.encryption-keyfile=/mytmpfs/mySecretKey\

--server.storage-engine=rocksdb

Thefile/mytmpfs/mySecretKeymustcontaintheencryptionkey.Thisfilemustbesecured,sothatonlyarangodcanaccessit.Youshouldalsoensurethatincasesome-onestealsthehardware,hewillnotbeabletoreadthefile.Forexample,byencryption/mytmpfsorcreatingain-memoryfile-systemunder/mytmpfs.

Encryptionkeygeneratedbyaprogram

Passthefollowingoptiontoarangod:

$arangod\

--rocksdb.encryption-key-generator=path-to-my-generator\

--server.storage-engine=rocksdb

Theprogrampath-to-my-generatoroutputtheencryptiononstandardoutputandexit.

Encryption

463

Creatingkeys

Theencryptionkeyfilemustcontain32bytesofrandomdata.

Youcancreateitwithacommandlinethis.

ddif=/dev/randombs=1count=32of=yourSecretKeyFile

Forsecurity,itisbesttocreatethesekeysoffline(awayfromyourdatabaseservers)anddirectlystoretheminyousecretmanagementtool.

Encryption

464

AuditingThisfeatureisavailableintheEnterpriseEdition.

Auditingallowsyoutomonitoraccesstothedatabaseindetail.Ingeneralauditlogsareoftheform

2016-01-0112:00:00|server|username|database|client-ip|authentication|text1|text2|...

Thetime-stampisinGMT.Thisallowstoeasilymatchlogentriesfromserversindifferenttimezones.

Thenameoftheserver.Youcanspecifyacustomnameonstartup.Otherwisethedefaulthostnameisused.

Theusernameisthe(authenticatedorunauthenticated)namesuppliedbytheclient.Adash-isprintedifnonamewasgivenbytheclient.

Thedatabasedescribesthedatabasethatwasaccessed.Pleasenotethattherearenodatabasecrossingqueries.Eachaccessisrestrictedtoonedatabase.

Theclient-ipdescribesthesourceoftherequest.

Theauthenticationdetailsthemethodsusedtoauthenticatetheuser.

Detailsabouttherequestsfollowintheadditionalfields.

Auditing

465

AuditConfigurationThisfeatureisavailableintheEnterpriseEdition.

Output--audit.outputoutput

Specifiesthetargetoftheauditlog.Possiblevaluesare

file://filenamewherefilenamecanberelativeorabsolute.

syslog://facilityorsyslog://facility/application-nametologintoasyslogserver.

Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputformultipletargets.

Hostname--audit.hostnamename

Thenameoftheserverusedinauditlogmessages.Bydefaultthesystemhostnameisused.

Configuration

466

Dropadatabase

2016-10-0415:33:25|server1|user1|database1|127.0.0.1:56920|httpbasic|deletedatabase'database1'|ok|/_api/dat

abase

Collections

Createacollection

2016-10-0517:35:57|server1|user1|database1|127.0.0.1:51294|httpbasic|createcollection'collection1'|ok|/_api

/collection

Truncateacollection

2016-10-0517:36:08|server1|user1|database1|127.0.0.1:51294|httpbasic|truncatecollection'collection1'|ok|/_a

pi/collection/collection1/truncate

Dropacollection

2016-10-0517:36:30|server1|user1|database1|127.0.0.1:51294|httpbasic|deletecollection'collection1'|ok|/_api

/collection/collection1

Indexes

Createaindex

2016-10-0518:19:40|server1|user1|database1|127.0.0.1:52467|httpbasic|createindexin'collection1'|ok|{"field

s":["a"],"sparse":false,"type":"skiplist","unique":false}|/_api/index?collection=collection1

Dropaindex

2016-10-0518:18:28|server1|user1|database1|127.0.0.1:52464|httpbasic|dropindex':44051'|ok|/_api/index/colle

ction1/44051

Documents

Readingasingledocument

2016-10-0412:27:55|server1|user1|database1|127.0.0.1:53699|httpbasic|createdocumentok|/_api/document/collecti

on1

Replacingasingledocument

2016-10-0412:28:08|server1|user1|database1|127.0.0.1:53699|httpbasic|replacedocumentok|/_api/document/collect

ion1/21456?ignoreRevs=false

Modifyingasingledocument

2016-10-0412:28:15|server1|user1|database1|127.0.0.1:53699|httpbasic|modifydocumentok|/_api/document/collecti

on1/21456?keepNull=true&ignoreRevs=false

Events

468

IntroductiontoReplicationReplicationallowsyoutoreplicatedataontoanothermachine.ItformsthebaseofalldisasterrecoveryandfailoverfeaturesArangoDBoffers.

ArangoDBoffersasynchronousandsynchronousreplication,dependingonwhichtypeofarangodbdeploymentyouareusing.SinceArangoDB3.2thesynchronousreplicationreplicationistheonlyreplicationtypeusedinaclusterwhereastheasynchronousreplicationisonlyavailablebetweensingle-servernodes.FutureversionsofArangoDBmayreintroduceasynchronousreplicationforthecluster.

Wewilldescribeprosandconsofeachoftheminthefollowingsections.

Asynchronousreplication

InArangoDBanywriteoperationwillbeloggedtothewrite-aheadlog.WhenusingAsynchronousreplicationslaveswillconnecttoamasterandapplyalltheeventsfromtheloginthesameorderlocally.Afterthat,theywillhavethesamestateofdataasthemasterdatabase.

Synchronousreplication

Synchronousreplicationonlyworkswithinaclusterandistypicallyusedformissioncriticaldatawhichmustbeaccessibleatalltimes.Synchronousreplicationgenerallystoresacopyofashard'sdataonanotherdbserverandkeepsitinsync.Essentially,whenstoringdataafterenablingsynchronousreplicationtheclusterwillwaitforallreplicastowriteallthedatabeforegreenlightingthewriteoperationtotheclient.Thiswillnaturallyincreasethelatencyabit,sinceonemorenetworkhopisneededforeachwrite.However,itwillenabletheclustertoimmediatelyfailovertoareplicawheneveranoutagehasbeendetected,withoutlosinganycommitteddata,andmostlywithoutevensignalinganerrorconditiontotheclient.

Synchronousreplicationisorganizedsuchthateveryshardhasaleaderandr-1followers,whererdenotedthereplicationfactor.ThenumberoffollowerscanbecontrolledusingthereplicationFactorparameterwheneveryoucreateacollection,thereplicationFactorparameteristhetotalnumberofcopiesbeingkept,thatis,itisoneplusthenumberoffollowers.

Satellitecollections

SatellitecollectionsaresynchronouslyreplicatedcollectionshavingadynamicreplicationFactor.Theywillreplicatealldatatoalldatabaseserversallowingthedatabaseserverstojoindatalocallyinsteadofdoingheavynetworkoperations.

Satellitecollectionsareanenterpriseonlyfeature.

Replication

470

AsynchronousreplicationAsynchronousreplicationworksbyloggingeverydatamodificationonamasterandreplayingtheseeventsonanumberofslaves.

Transactionsarehonoredinreplication,i.e.transactionalwriteoperationswillbecomevisibleonslavesatomically.

Asallwriteoperationswillbeloggedtoamasterdatabase'swrite-aheadlog,thereplicationinArangoDBcurrentlycannotbeusedforwrite-scaling.ThemainpurposesofthereplicationincurrentArangoDBaretoprovideread-scalabilityand"hotbackups"ofspecificdatabases.

Itispossibletoconnectmultipleslavedatabasestothesamemasterdatabase.Slavedatabasesshouldbeusedasread-onlyinstances,andnouser-initiatedwriteoperationsshouldbecarriedoutonthem.Otherwisedataconflictsmayoccurthatcannotbesolvedautomatically,andthatwillmakethereplicationstop.

Inanasynchronousreplicationscenarioslaveswillpullchangesfromthemasterdatabase.Slavesneedtoknowtowhichmasterdatabasetheyshouldconnectto,butamasterdatabaseisnotawareoftheslavesthatreplicatefromit.Whenthenetworkconnectionbetweenthemasterdatabaseandaslavegoesdown,writeoperationsonthemastercancontinuenormally.Whenthenetworkisupagain,slavescanreconnecttothemasterdatabaseandtransfertheremainingchanges.Thiswillhappenautomaticallyprovidedslavesareconfiguredappropriately.

Replicationlag

Inthissetup,writeoperationsareappliedfirstinthemasterdatabase,andappliedintheslavedatabase(s)afterwards.

Forexample,let'sassumeawriteoperationisexecutedinthemasterdatabaseatpointintimet0.Tomakeaslavedatabaseapplythesameoperation,itmustfirstfetchthewriteoperation'sdatafrommasterdatabase'swrite-aheadlog,thenparseitandapplyitlocally.Thiswillhappenatsomepointintimeaftert0,let'ssayt1.

Thedifferencebetweent1andt0iscalledthereplicationlag,anditisunavoidableinasynchronousreplication.Theamountofreplicationlagdependsonmanyfactors,afewofwhichare:

thenetworkcapacitybetweentheslavesandthemastertheloadofthemasterandtheslavesthefrequencyinwhichslavespollthemasterforupdates

Betweent0andt1,thestateofdataonthemasterisnewerthanthestateofdataontheslave(s).Atpointintimet1,thestateofdataonthemasterandslave(s)isconsistentagain(providednonewdatamodificationshappenedonthemasterinbetween).Thus,thereplicationwillleadtoaneventuallyconsistentstateofdata.

Replicationconfiguration

Thereplicationisturnedoffbydefault.Inordertocreateamaster-slavesetup,theso-calledreplicationapplierneedstobeenabledontheslavedatabases.

Replicationisconfiguredonaper-databaselevel.Ifmultipledatabasearetobereplicated,thereplicationmustbesetupindividuallyperdatabase.

Thereplicationapplierontheslavecanbeusedtoperformaone-timesynchronizationwiththemaster(andthenstop),ortoperformanongoingreplicationofchanges.Toresumereplicationonslaverestart,theautoStartattributeofthereplicationappliermustbesettotrue.

Replicationoverhead

Asthemasterserversarelogginganywriteoperationinthewrite-ahead-loganywayreplicationdoesn'tcauseanyextraoverheadonthemaster.Howeveritwillofcoursecausesomeoverheadforthemastertoserveincomingreadrequestsoftheslaves.Returningtherequesteddataishoweveratrivialtaskforthemasterandshouldnotresultinanotableperformancedegrationinproduction.


471

Components

ReplicationLogger

Purpose

Thereplicationloggerwillwritealldata-modificationoperationsintothewrite-aheadlog.Thislogmaythenbereadbyclientstoreplayanydatamodificationonadifferentserver.

Checkingthestate

Toquerythecurrentstateofthelogger,usethestatecommand:

require("@arangodb/replication").logger.state();

Theresultmightlooklikethis:

{

"state":{

"running":true,

"lastLogTick":"133322013",

"totalEvents":16,

"time":"2014-07-06T12:58:11Z"

},

"server":{

"version":"2.2.0-devel",

"serverId":"40897075811372"

},

"clients":{

}

}

Therunningattributewillalwaysbetrue.InearlierversionsofArangoDBthereplicationwasoptionalandthiscouldhavebeenfalse.

ThetotalEventsattributeindicateshowmanylogeventshavebeenloggedsincethestartoftheArangoDBserver.Finally,thelastLogTickvalueindicatestheidofthelastoperationthatwaswrittentotheserver'swrite-aheadlog.Itcanbeusedtodeterminewhethernewoperationswerelogged,andisalsousedbythereplicationapplierforincrementalfetchingofdata.

Note:ThereplicationloggerstatecanalsobequeriedviatheHTTPAPI.

Toquerywhichdatarangesarestillavailableforreplicationclientstofetch,theloggerprovidesthefirstTickandtickRangesfunctions:

require("@arangodb/replication").logger.firstTick();

ThiswillreturntheminimumtickvaluethattheservercanprovidetoreplicationclientsviaitsreplicationAPIs.ThetickRangesfunctionreturnstheminimumandmaximumtickvaluesperlogfile:

require("@arangodb/replication").logger.tickRanges();

ReplicationApplier

Purpose

Thepurposeofthereplicationapplieristoreaddatafromamasterdatabase'seventlog,andapplythemlocally.Theapplierwillcheckthemasterdatabasefornewoperationsperiodically.Itwillperformanincrementalsynchronization,i.e.onlyaskingthemasterforoperationsthatoccurredafterthelastsynchronization.


472

Thereplicationapplierdoesnotgetnotifiedbythemasterdatabasewhenthereare"new"operationsavailable,butinsteadusesthepullprinciple.Itmightthustakesometime(theso-calledreplicationlag)beforeanoperationfromthemasterdatabasegetsshippedtoandappliedinaslavedatabase.

Thereplicationapplierofadatabaseisruninaseparatethread.Itmayencounterproblemswhenanoperationfromthemastercannotbeappliedsafely,orwhentheconnectiontothemasterdatabasegoesdown(networkoutage,masterdatabaseisdownorunavailableetc.).Inthiscase,thedatabase'sreplicationapplierthreadmightterminateitself.Itisthenuptotheadministratortofixtheproblemandrestartthedatabase'sreplicationapplier.

Ifthereplicationappliercannotconnecttothemasterdatabase,orthecommunicationfailsatsomepointduringthesynchronization,thereplicationapplierwilltrytoreconnecttothemasterdatabase.Itwillgiveupreconnectingonlyafteraconfigurableamountofconnectionattempts.

Thereplicationapplierstateisqueryableatanytimebyusingthestatecommandoftheapplier.Thiswillreturnthestateoftheapplierofthecurrentdatabase:

require("@arangodb/replication").applier.state();


{

"state":{

"running":true,

"lastAppliedContinuousTick":"152786205",

"lastProcessedContinuousTick":"152786205",

"lastAvailableContinuousTick":"152786205",

"progress":{

"time":"2014-07-06T13:04:57Z",

"message":"fetchingmasterlogfromoffset152786205",

"failedConnects":0

},

"totalRequests":38,

"totalFailedConnects":0,

"totalEvents":1,

"lastError":{

"errorNum":0

},

"time":"2014-07-06T13:04:57Z"

},

"server":{

"version":"2.2.0-devel",

"serverId":"210189384542896"

},

"endpoint":"tcp://master.example.org:8529",

"database":"_system"

}

Therunningattributeindicateswhetherthereplicationapplierofthecurrentdatabaseiscurrentlyrunningandpollingtheserveratendpointfornewevents.

Theprogress.failedConnectsattributeshowshowmanyfailedconnectionattemptsthereplicationappliercurrentlyhasencounteredinarow.Incontrast,thetotalFailedConnectsattributeindicateshowmanyfailedconnectionattemptstheapplierhasmadeintotal.ThetotalRequestsattributeshowshowmanyrequeststheapplierhassenttothemasterdatabaseintotal.ThetotalEventsattributeshowshowmanylogeventstheapplierhasreadfromthemaster.

Theprogress.messagesub-attributeprovidesabriefhintofwhattheappliercurrentlydoes(ifitisrunning).ThelastErrorattributealsohasanoptionalerrorMessagesub-attribute,showingthelatesterrormessage.TheerrorNumsub-attributeofthelastErrorattributecanbeusedbyclientstoprogrammaticallycheckforerrors.Itshouldbe0ifthereisnoerror,anditshouldbenon-zeroiftheapplierterminateditselfduetoaproblem.

Hereisanexampleofthestateafterthereplicationapplierterminateditselfdueto(repeated)connectionproblems:

{

"state":{

"running":false,

"progress":{

"time":"2014-07-06T13:14:37Z",


473

"message":"applierstopped",

"failedConnects":6

},

"totalRequests":79,

"totalFailedConnects":11,

"totalEvents":0,

"lastError":{

"time":"2014-07-06T13:09:41Z",

"errorMessage":"couldnotconnecttomasterattcp://master.example.org:8529:Couldnotconnectto'tcp:/...",

"errorNum":1400

},

...

}

}

Note:thestateofadatabase'sreplicationapplierisqueryableviatheHTTPAPI,too.PleaserefertoHTTPInterfaceforReplicationformoredetails.

All-in-onesetup

Tocopytheinitialdatafromtheslavetothemasterandstartthecontinuousreplication,thereisanall-in-onecommandsetupReplication:

require("@arangodb/replication").setupReplication(configuration);

Thefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthe_systemdatabase.Notethatitshouldberunontheslaveandnotthemaster:

db._useDatabase("_system");

require("@arangodb/replication").setupReplication({

endpoint:"tcp://master.domain.org:8529",

username:"myuser",

password:"mypasswd",

verbose:false,

includeSystem:false,

incremental:true,

autoResync:true

});

Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationisstarted,orincasetheinitialsynchronizationhasfailed.

Iftheinitialsynchronizationissuccessful,thecommandwillstorethegivenconfigurationontheslave.Italsoconfiguresthecontinuousreplicationtostartautomaticallyiftheslaveisrestarted,i.e.autoStartissettotrue.

Ifthecommandisrunwhiletheslave'sreplicationapplierisalreadyrunning,itwillfirststoptherunningapplier,dropitsconfigurationanddoaresynchronizationofdatawiththemaster.Itwillthenusetheprovidedconfigration,overwritinganypreviouslyexistingreplicationconfigurationontheslave.

StartingandStopping

Tomanuallystartandstoptheapplierinthecurrentdatabase,thestartandstopcommandscanbeusedlikethis:

require("@arangodb/replication").applier.start(<tick>);

require("@arangodb/replication").applier.stop();

Note:Startingareplicationapplierwithoutsettingupaninitialconfigurationwillfail.ThereplicationapplierwilllookforitsconfigurationinafilenamedREPLICATION-APPLIER-CONFIGinthecurrentdatabase'sdirectory.Ifthefileisnotpresent,ArangoDBwillusesomedefaultconfiguration,butitcannotguesstheendpoint(theaddressofthemasterdatabase)theappliershouldconnectto.Thusstartingtheapplierwithoutconfigurationwillfail.

Notethatatthefirsttimeyoustarttheapplier,youshouldpassthevaluereturnedinthelastLogTickattributeoftheinitialsyncoperation.


474

Note:Startingadatabase'sreplicationapplierviathestartcommandwillnotnecessarilystarttheapplieronthenextandfollowingArangoDBserverrestarts.Additionally,stoppingadatabase'sreplicationappliermanuallywillnotnecessarilypreventtheapplierfrombeingstartedagainonthenextserverstart.Allofthisisconfigurableseparately(hangonreading).

Note:whenstoppingandrestartingthereplicationapplierofdatabase,itwillresumewhereitlaststopped.Thisissensiblebecausereplicationlogeventsshouldbeappliedincrementally.Ifthereplicationapplierofadatabasehasneverbeenstartedbefore,itneedssometickvaluefromthemaster'slogfromwhichtostartfetchingevents.

Thereisonecaveattoconsiderwhenstoppingareplicationontheslave:iftherearestillongoingreplicatedtransactionsthatareneithercommittedoraborted,stoppingthereplicationapplierwillcausetheseoperationstobelostfortheslave.Ifthesetransactionscommitonthemasterlaterandthereplicationisresumed,theslavewillnotbeabletocommitthesetransactions,too.Thusstoppingthereplicationapplierontheslavemanuallyshouldonlybedoneifthereiscertaintythattherearenoongoingtransactionsonthemaster.

Configuration

Toconfigurethereplicationapplierofaspecificdatabase,usethepropertiescommand.Usingitwithoutanyargumentswillreturntheapplier'scurrentconfiguration:

require("@arangodb/replication").applier.properties();


{

"requestTimeout":600,

"connectTimeout":10,

"ignoreErrors":0,

"maxConnectRetries":10,

"chunkSize":0,

"autoStart":false,

"adaptivePolling":true,

"includeSystem":true,

"requireFromPresent":false,

"autoResync":false,

"autoResyncRetries":2,

"verbose":false

}

Note:Thereisnoendpointattributeconfiguredyet.Theendpointattributeisrequiredforthereplicationappliertobestartable.Youmayalsowanttoconfigureausernameandpasswordfortheconnectionviatheusernameandpasswordattributes.

require("@arangodb/replication").applier.properties({


username:"root",

password:"secret",

verbose:false

});

Thiswillre-configurethereplicationapplierforthecurrentdatabase.Theconfigurationwillbeusedfromthenextstartofthereplicationapplier.Thereplicationappliercannotbere-configuredwhileitisrunning.Itmustbestoppedfirsttobere-configured.

TomakethereplicationapplierofthecurrentdatabasestartautomaticallywhentheArangoDBserverstarts,usetheautoStartattribute.

SettingtheadaptivePollingattributetotruewillmakethereplicationapplierpollthemasterdatabaseforchangeswithavariablefrequency.Thereplicationapplierwillthenlowerthefrequencywhenthemasterisidle,andincreaseitwhenthemastercanprovidenewevents).Otherwisethereplicationapplierwillpollthemasterdatabaseforchangeswithaconstantfrequency.

TheidleMinWaitTimeattributecontrolstheminimumwaittime(inseconds)thatthereplicationapplierwillintentionallyidlebeforefetchingmorelogdatafromthemasterincasethemasterhasalreadysentallitslogdata.ThiswaittimecanbeusedtocontrolthefrequencywithwhichthereplicationappliersendsHTTPlogfetchrequeststothemasterincasethereisnowriteactivityonthemaster.

TheidleMaxWaitTimeattributecontrolsthemaximumwaittime(inseconds)thatthereplicationapplierwillintentionallyidlebeforefetchingmorelogdatafromthemasterincasethemasterhasalreadysentallitslogdataandtherehavebeenpreviouslogfetchattemptsthatresultedinnomorelogdata.Thiswaittimecanbeusedtocontrolthemaximumfrequencywithwhichthereplicationappliersends


475

HTTPlogfetchrequeststothemasterincasethereisnowriteactivityonthemasterforlongerperiods.NotethatthisconfigurationvaluewillonlybeusediftheoptionadaptivePollingissettotrue.

Tosetatimeoutforconnectionandfollowingrequestattempts,usetheconnectTimeoutandrequestTimeoutvalues.ThemaxConnectRetriesattributeconfiguresafterhowmanyfailedconnectionattemptsinarowthereplicationapplierwillgiveupandturnitselfoff.Youmaywanttosetthistoahighvaluesothattemporarynetworkoutagesdonotleadtothereplicationapplierstoppingitself.TheconnectRetryWaitTimeattributeconfigureshowlongthereplicationapplierwillwaitbeforeretryingtheconnectiontothemasterincaseofconnectionproblems.

ThechunkSizeattributecanbeusedtocontroltheapproximatemaximumsizeofamaster'sresponse(inbytes).Settingittoalowvaluemaymakethemasterrespondfaster(lessdataisassembledbeforethemastersendstheresponse),butmayrequiremorerequest-responseroundtrips.Setitto0touseArangoDB'sbuilt-indefaultvalue.

TheincludeSystemattributecontrolswhetherchangestosystemcollections(suchas_graphsor_users)shouldbeapplied.Ifsettotrue,changesinthesecollectionswillbereplicated,otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.

TherequireFromPresentattributecontrolswhethertheapplierwillstartsynchronizingincaseitdetectsthatthemastercannotprovidedatafortheinitialtickvalueprovidedbytheslave.ThismaybethecaseifthemasterdoesnothaveabigenoughbacklogofhistoricWALlogfiles,andwhenthereplicationisre-startedafteralongerpause.WhenrequireFromPresentissettotrue,thenthereplicationapplierwillcheckatstartwhetherthestarttickfromwhichitstartsorresumesreplicationisstillpresentonthemaster.Ifnot,thentherewouldbedataloss.IfrequireFromPresentistrue,thereplicationapplierwillabortwithanappropriateerrormessage.Ifsettofalse,thenthereplicationapplierwillstillstart,andignorethedataloss.

TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslaverequests,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.

Automaticre-synchronizationmaytransferalotofdatafromthemastertotheslaveandcanbeexpensive.Itisthereforeturnedoffbydefault.Whenturnedoff,theslavewillneverperformanautomaticre-synchronizationwiththemaster.

TheautoResyncRetriesoptioncanbeusedtocontrolthenumberofresynchronizationretriesthatwillbeperformedinarowwhenautomaticresynchronizationisenabledandkicksin.Settingthisto0willeffectivelydisableautoResync.Settingittosomeothervaluewilllimitthenumberofretriesthatareperformed.Thishelpspreventingendlessretriesincaseresynchronizationsalwaysfail.

Theverboseattributecontrolstheverbosityofthereplicationlogger.Settingittotruewillmakethereplicationapplierwritealinetothelogforeveryoperationitperforms.Thisshouldonlybeusedfordiagnosingreplicationproblems.

Thefollowingexamplewillsetmostofthediscussedpropertiesforthecurrentdatabase'sapplier:



username:"root",

password:"secret",

adaptivePolling:true,

connectTimeout:15,

maxConnectRetries:100,

chunkSize:262144,

autoStart:true,

includeSystem:true,

autoResync:true,

autoResyncRetries:2,

});

Aftertheapplierisnowfullyconfigured,itcouldtheoreticallybestarted.However,wemayfirstneedaninitialsynchronizationofallcollectionsandtheirdatafromthemasterbeforewestartthereplicationapplier.

Theonlysafemethodfordoingafullsynchronization(orre-synchronization)isthusto

stopthereplicationapplierontheslave(ifcurrentlyrunning)performaninitialfullsyncwiththemasterdatabasenotethemasterdatabase'slastLogTickvalueand


476

startthecontinuousreplicationapplierontheslaveusingthistickvalue.

Theinitialsynchronizationforthecurrentdatabaseisexecutedwiththesynccommand:

require("@arangodb/replication").sync({


username:"root",

password:"secret,

includeSystem:true

});

TheincludeSystemoptioncontrolswhetherdatafromsystemcollections(suchas_graphsand_users)shallbesynchronized.

TheinitialsynchronizationcanoptionallybeconfiguredtoincludeorexcludespecificcollectionsusingtherestrictTypeandrestrictCollectionparameters.

Thefollowingcommandonlysynchronizescollectionfooandbar:



username:"root",

password:"secret,

restrictType:"include",

restrictCollections:["foo","bar"]

});

UsingarestrictTypeofexclude,allcollectionsbutthespecifiedwillbesynchronized.

Warning:syncwilldoafullsynchronizationofthecollectionsinthecurrentdatabasewithcollectionspresentinthemasterdatabase.Anylocalinstancesofthecollectionsandalltheirdataareremoved!Onlyexecutethiscommandifyouaresureyouwanttoremovethelocaldata!

Assyncdoesafullsynchronization,itmighttakeawhiletoexecute.Whensynccompletessuccessfully,itreturnsanarrayofcollectionsithassynchronizedinitscollectionsattribute.Itwillalsoreturnthemasterdatabase'slastlogtickvalueatthetimethesyncwasstartedonthemaster.ThetickvalueiscontainedinthelastLogTickattributeofthesynccommand:

{

"lastLogTick":"231848833079705",

"collections":[...]

}

Nowyoucanstartthecontinuoussynchronizationforthecurrentdatabaseontheslavewiththecommand

require("@arangodb/replication").applier.start("231848833079705");

Note:Thetickvaluesshouldbetreatedasstrings.Usingnumericdatatypesfortickvaluesisunsafebecausetheymightexceedthe32bitvalueandtheIEEE754doubleaccuracyranges.


477

Per-DatabaseSetupThispagedescribesthereplicationprocessbasedonaspecificdatabasewithinanArangoDBinstance.Thatmeansthatonlythespecifieddatabasewillbereplicated.

Settingupaworkingmaster-slavereplicationrequirestwoArangoDBinstances:

master:thisistheinstancethatalldata-modificationoperationsshouldbedirectedtoslave:onthisinstance,we'llstartareplicationapplier,andthiswillfetchdatafromthemasterdatabase'swrite-aheadlogandapplyitsoperationslocally

Forthefollowingexamplesetup,we'llusetheinstancetcp://master.domain.org:8529asthemaster,andtheinstancetcp://slave.domain.org:8530asaslave.

Thegoalistohavealldatafromthedatabase_systemonmastertcp://master.domain.org:8529bereplicatedtothedatabase_systemontheslavetcp://slave.domain.org:8530.

Onthemaster,nothingspecialneedstobedone,asallwriteoperationswillautomaticallybeloggedinthemaster'swrite-aheadlog(WAL).

All-in-onesetup

Tomakethereplicationcopytheinitialdatafromthemastertotheslaveandstartthecontinuousreplicationontheslave,thereisanall-in-onecommand:

require("@arangodb/replication").setupReplication(configuration);

Thefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthe_systemdatabase.Notethatitshouldberunontheslaveandnotthemaster:


require("@arangodb/replication").setupReplication({


username:"myuser",


verbose:false,


incremental:true,

autoResync:true

});

Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationhasbeenstarted,orincasetheinitialsynchronizationhasfailed.



Initialsynchronization

Theinitialsynchronizationandcontinuousreplicationappliercanalsobestartedseparately.Tostartreplicationontheslave,makesuretherecurrentlyisnoreplicationapplierrunning.

Thefollowingcommandsstoparunningapplierintheslave's_systemdatabase:



478

require("@arangodb/replication").applier.stop();

Thestopoperationwillterminateanyreplicationactivityinthe_systemdatabaseontheslave.

Afterthat,theinitialsynchronizationcanberun.Itwillcopythecollectionsfromthemastertotheslave,overwritingexistingdata.Toruntheinitialsynchronization,executethefollowingcommandsontheslave:




username:"myuser",


verbose:false

});

Usernameandpasswordonlyneedtobespecifiedwhenthemasterrequiresauthentication.Tocheckwhatthesynchronizationiscurrentlydoing,supplysettheverboseoptiontotrue.Ifset,thesynchronizationwillcreatelogmessageswiththecurrentsynchronizationstatus.

Warning:Thesynccommandwillreplacedataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!

ThesyncoperationwillreturnanattributenamedlastLogTickwhichwe'llneedtonote.Thelastlogtickwillbeusedasthestartingpointforsubsequentreplicationactivity.Let'sassumewegotthefollowinglastlogtick:

{


...

}

InitialsynchronizationfromtheArangoShell

Theinitialsynchronizationviathesynccommandmaytakealongtimetocomplete.Theshellwillblockuntiltheslavehascompletedtheinitialsynchronizationoruntilanerroroccurs.Bydefault,thesynccommandintheArangoShellwillpolltheslaveforastatusupdateevery10seconds.

Optionallythesynccommandcanbemadenon-blockingbysettingitsasyncoptiontotrue.Inthiscase,thesynccommandwillreturninstantlywithanidstring,andtheinitialsynchronizationwillrundetachedonthemaster.TofetchthecurrentstatusofthesyncprogressfromtheArangoShell,thegetSyncResultfunctioncanbeusedasfollows:


varreplication=require("@arangodb/replication");

/*runcommandinasyncmode*/

varid=replication.sync({


username:"myuser",


async:true

});

/*nowquerythestatusofouroperation*/

print(replication.getSyncResult(id));

getSyncResultwillreturnfalseaslongasthesynchronizationisnotcomplete,andreturnthesynchronizationresultotherwise.

Continuoussynchronization

Whentheinitialsynchronizationisfinished,thecontinuousreplicationappliercanbestartedusingthelastlogtickprovidedbythesynccommand.Beforestartingit,thereisatleastoneconfigurationoptiontoconsider:replicationontheslavewillberunninguntiltheslavegetsshutdown.Whentheslaveservergetsrestarted,replicationwillbeturnedoffagain.Tochangethis,wefirstneedtoconfiguretheslave'sreplicationapplierandsetitsautoStartattribute.


479

Here'sthecommandtoconfigurethereplicationapplierwithseveraloptions,includingtheautoStartattribute:




username:"myuser",


autoStart:true,

autoResync:true,




requireFromPresent:false,

idleMinWaitTime:0.5,

idleMaxWaitTime:1.5,

verbose:false

});

Animportantconsiderationforreplicationiswhetherdatafromsystemcollections(suchas_graphsor_users)shouldbeapplied.TheincludeSystemoptioncontrolsthat.Ifsettotrue,changesinsystemcollectionswillbereplicated.Otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.


TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslavehadrequested,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.

Notethatautomaticre-synchronization(autoResyncoptionsettotrue)maytransferalotofdatafromthemastertotheslaveandcanthereforebeexpensive.Stillit'sturnedonheresothere'slessneedformanualintervention.


Nowit'stimetostartthereplicationapplierontheslaveusingthelastlogtickwegotbefore:


require("@arangodb/replication").applier.start("40694126");

Thiswillreplicatealloperationshappeninginthemaster'ssystemdatabaseandapplythemontheslave,too.

Afterthat,youshouldbeabletomonitorthestateandprogressofthereplicationapplierbyexecutingthestatecommandontheslaveserver:


require("@arangodb/replication").applier.state();

Pleasenotethatstoppingthereplicationapplierontheslaveusingthestopcommandshouldbeavoided.Thereasonisthatcurrentlyongoingtransactions(thathavepartlybeenreplicatedtotheslave)willbeneedtoberestartedafterarestartofthereplicationapplier.Stoppingandrestartingthereplicationapplierontheslaveshouldthusonlybeperformedifthereiscertaintythatthemasteriscurrentlyfullyidleandalltransactionshavebeenreplicatedfully.

Notethatwhileaslavehasonlypartlyexecutedatransactionfromthemaster,itmightkeepawritelockonthecollectionsinvolvedinthetransaction.


480

YoumayalsowanttocheckthemasterandslavestatesviatheHTTPAPIs(seeHTTPInterfaceforReplication).


481

Server-levelSetupThispagedescribesthereplicationprocessbasedonacompleteArangoDBinstance.Thatmeansthatallincludeddatabaseswillbereplicated.

Settingupaworkingmaster-slavereplicationrequirestwoArangoDBinstances:

master:thisistheinstancethatalldata-modificationoperationsshouldbedirectedtoslave:onthisinstance,we'llstartareplicationapplier,andthiswillfetchdatafromthemasterdatabase'swrite-aheadlogandapplyitsoperationslocally


Thegoalistohavealldataofalldatabasesonmastertcp://master.domain.org:8529bereplicatedtotheslaveinstancetcp://slave.domain.org:8530.

Onthemaster,nothingspecialneedstobedone,asallwriteoperationswillautomaticallybeloggedinthemaster'swrite-aheadlog(WAL).

All-in-onesetup

Tomakethereplicationcopytheinitialdatafromthemastertotheslaveandstartthecontinuousreplicationontheslave,thereisanall-in-onecommand:

require("@arangodb/replication").setupReplicationGlobal(configuration);

ThefollowingexampledemonstrateshowtousethecommandforsettingupreplicationforthecompleteArangoDBinstance.Notethatitshouldberunontheslaveandnotthemaster:


require("@arangodb/replication").setupReplicationGlobal({

endpoint:"tcp://127.0.0.1:8529",

username:"root",

password:"",

autoStart:true

});

Thecommandwillreturnwhentheinitialsynchronizationisfinishedandthecontinuousreplicationhasbeenstarted,orincasetheinitialsynchronizationhasfailed.



Stoppingsynchronization

Theinitialsynchronizationandcontinuousreplicationappliercanalsobestartedseparately.Tostartreplicationontheslave,makesuretherecurrentlyisnoreplicationapplierrunning.

Thefollowingcommandsstoparunningapplierintheslave'sinstance:


require("@arangodb/replication").globalApplier.stop();


482

ThestopoperationwillterminateanyreplicationactivityintheArangoDBinstanceontheslave.

Afterthat,theinitialsynchronizationcanberun.Itwillcopythecollectionsfromthemastertotheslave,overwritingexistingdata.Toruntheinitialsynchronization,executethefollowingcommandsontheslave:


require("@arangodb/replication").syncGlobal({


username:"myuser",


verbose:false

});

Usernameandpasswordonlyneedtobespecifiedwhenthemasterrequiresauthentication.Tocheckwhatthesynchronizationiscurrentlydoing,supplysettheverboseoptiontotrue.Ifset,thesynchronizationwillcreatelogmessageswiththecurrentsynchronizationstatus.

Warning:ThesyncGlobalcommandwillreplacedataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!

ThesyncoperationwillreturnanattributenamedlastLogTickwhichwe'llneedtonote.Thelastlogtickwillbeusedasthestartingpointforsubsequentreplicationactivity.Let'sassumewegotthefollowinglastlogtick:

{


...

}

InitialsynchronizationfromtheArangoShellTheinitialsynchronizationviathesyncGlobalcommandmaytakealongtimetocomplete.Theshellwillblockuntiltheslavehascompletedtheinitialsynchronizationoruntilanerroroccurs.Bydefault,thesyncGlobalcommandintheArangoShellwillpolltheslaveforastatusupdateevery10seconds.

OptionallythesyncGlobalcommandcanbemadenon-blockingbysettingitsasyncoptiontotrue.Inthiscase,thesyncGlobalcommandwillreturninstantlywithanidstring,andtheinitialsynchronizationwillrundetachedonthemaster.TofetchthecurrentstatusofthesyncGlobalprogressfromtheArangoShell,thegetSyncResultfunctioncanbeusedasfollows:




varid=replication.syncGlobal({


username:"myuser",


async:true

});




Continuoussynchronization

Whentheinitialsynchronizationisfinished,thecontinuousreplicationappliercanbestartedusingthelastlogtickprovidedbythesyncGlobalcommand.Beforestartingit,thereisatleastoneconfigurationoptiontoconsider:replicationontheslavewillberunninguntiltheslavegetsshutdown.Whentheslaveservergetsrestarted,replicationwillbeturnedoffagain.Tochangethis,wefirstneedtoconfiguretheslave'sreplicationapplierandsetitsautoStartattribute.

Here'sthecommandtoconfigurethereplicationapplierwithseveraloptions,includingtheautoStartattribute:


483


require("@arangodb/replication").globalApplier.properties({


username:"myuser",


autoStart:true,

autoResync:true,




requireFromPresent:false,

idleMinWaitTime:0.5,

idleMaxWaitTime:1.5,

verbose:false

});

Animportantconsiderationforreplicationiswhetherdatafromsystemcollections(suchas_graphsor_users)shouldbeapplied.TheincludeSystemoptioncontrolsthat.Ifsettotrue,changesinsystemcollectionswillbereplicated.Otherwise,theywillnotbereplicated.Itisoftennotnecessarytoreplicatedatafromsystemcollections,especiallybecauseitmayleadtoconfusionontheslavebecausetheslaveneedstohaveitsownsystemcollectionsinordertostartandkeepoperational.


TheautoResyncoptioncanbeusedinconjunctionwiththerequireFromPresentoptionasfollows:whenbothrequireFromPresentandautoResyncaresettotrueandthemastercannotprovidethelogdatatheslavehadrequested,thereplicationapplierwillstopasusual.ButduetothefactthatautoResyncissettotrue,theslavewillautomaticallytriggerafullresyncofalldatawiththemaster.Afterthat,thereplicationapplierwillgointocontinuousreplicationmodeagain.Additionally,settingautoResynctotruewilltriggerafullre-synchronizationofdatawhenthecontinuousreplicationisstartedanddetectsthatthereisnostarttickvalue.

Notethatautomaticre-synchronization(autoResyncoptionsettotrue)maytransferalotofdatafromthemastertotheslaveandcanthereforebeexpensive.Stillit'sturnedonheresothere'slessneedformanualintervention.


Nowit'stimetostartthereplicationapplierontheslaveusingthelastlogtickwegotbefore:


require("@arangodb/replication").globalApplier.start("40694126");

Thiswillreplicatealloperationshappeninginthemaster'ssystemdatabaseandapplythemontheslave,too.

Afterthat,youshouldbeabletomonitorthestateandprogressofthereplicationapplierbyexecutingthestatecommandontheslaveserver:


require("@arangodb/replication").globalApplier.state();

Pleasenotethatstoppingthereplicationapplierontheslaveusingthestopcommandshouldbeavoided.Thereasonisthatcurrentlyongoingtransactions(thathavepartlybeenreplicatedtotheslave)willbeneedtoberestartedafterarestartofthereplicationapplier.Stoppingandrestartingthereplicationapplierontheslaveshouldthusonlybeperformedifthereiscertaintythatthemasteriscurrentlyfullyidleandalltransactionshavebeenreplicatedfully.

Notethatwhileaslavehasonlypartlyexecutedatransactionfromthemaster,itmightkeepawritelockonthecollectionsinvolvedinthetransaction.

YoumayalsowanttocheckthemasterandslavestatesviatheHTTPAPIs(seeHTTPInterfaceforReplication).


484


485

SyncingCollectionsInordertosynchronizedataforasinglecollectionfromamastertoaslaveinstance,thereisthesyncCollectionfunction:

Itwillfetchalldocumentsofthespecifiedcollectionfromthemasterdatabaseandstoretheminthelocalinstance.Afterthesynchronization,thecollectiondataontheslavewillbeidenticaltothedataonthemaster,providednofurtherdatachangeshappenonthemaster.AnydatachangesthatareperformedonthemasterafterthesynchronizationwasstartedwillnotbecapturedbysyncCollection,butneedtobereplicatedusingtheregularreplicationappliermechanism.


Thegoalistohavealldatafromthecollectiontestindatabase_systemonmastertcp://master.domain.org:8529bereplicatedtothecollectiontestindatabase_systemontheslavetcp://slave.domain.org:8530.

Onthemaster,thecollectiontestneedstobepresentinthe_systemdatabase,withanydatainit.

Totransferthiscollectiontotheslave,issuethefollowingcommandsthere:


require("@arangodb/replication").syncCollection("test",{


username:"myuser",

password:"mypasswd"

});

Warning:ThesyncCollectioncommandwillreplacethecollection'sdataintheslavedatabasewithdatafromthemasterdatabase!Onlyexecutethesecommandsifyouhaveverifiedyouareonthecorrectserver,inthecorrectdatabase!

SettingtheoptionalincrementalattributeinthecalltosyncCollectionwillstartanincrementaltransferofdata.Thismaybeusefulincasewhentheslavealreadyhaspartsoralmostallofthedatainthecollectionandonlythedifferencesneedtobesynchronized.Notethattocomputethedifferencestheincrementaltransferwillbuildasortedlistofalldocumentkeysinthecollectiononboththeslaveandthemaster,whichmaystillbeexpensiveforhugecollectionsintermsofmemoryusageandruntime.Duringbuildingthelistofkeysthecollectionwillberead-lockedonthemaster.

TheinitialSyncMaxWaitTimeattributeinthecalltosyncCollectioncontrolshowlongtheslavewillwaitforamaster'sresponse.Thiswaittimecanbeusedtocontrolafterwhattimethesynchronizationwillgiveupandfail.

ThesyncCollectioncommandmaytakealongtimetocompleteifthecollectionisbig.Theshellwillblockuntiltheslavehassynchronizedtheentirecollectionfromthemasteroruntilanerroroccurs.Bydefault,thesyncCollectioncommandintheArangoShellwillpollforastatusupdateevery10seconds.

WhensyncCollectioniscalledfromtheArangoShell,theoptionalasyncattributecanbeusedtostartthesynchronizationasabackgroundprocessontheslave.Ifasyncissettotrue,thecalltosyncCollectionwillreturnalmostinstantlywithanidstring.Usingthisidstring,thestatusofthesyncjobontheslavecanbequeriedusingthegetSyncResultfunctionasfollows:




varid=replication.syncCollection("test",{


username:"myuser",


async:true

});





486


487

ReplicationLimitationsThereplicationinArangoDBhasafewlimitations.SomeoftheselimitationsmayberemovedinlaterversionsofArangoDB:

thereisnofeedbackfromtheslavestothemaster.Ifaslavecannotapplyaneventitgotfromthemaster,themasterwillhaveadifferentstateofdata.Inthiscase,thereplicationapplierontheslavewillstopandreportanerror.Administratorscantheneither"fix"theproblemorre-syncthedatafromthemastertotheslaveandstarttheapplieragain.atthemomentitisassumedthatonlythereplicationapplierexecuteswriteoperationsonaslave.ArangoDBcurrentlydoesnotpreventusersfromcarryingouttheirownwriteoperationsonslaves,thoughthismightleadtoundefinedbehaviorandthereplicationapplierstopping.whenareplicationslaveasksamasterforlogevents,thereplicationmasterwillreturnallwriteoperationsforuser-definedcollections,butitwillexcludewriteoperationsforcertainsystemcollections.Thefollowingcollectionsareexcludedintentionallyfromreplication:_apps,_trx,_replication,_configuration,_jobs,_queues,_sessions,_foxxlogandallstatisticscollections.Writeoperationsforthefollowingsystemcollectionscanbequeriedfromamaster:_aqlfunctions,_graphs,_users.Foxxapplicationsconsistofdatabaseentriesandapplicationscriptsinthefilesystem.ThefilesystempartsofFoxxapplicationsarenottrackedanywhereandthusnotreplicatedincurrentversionsofArangoDB.ToreplicateaFoxxapplication,itisrequiredtocopytheapplicationtotheremoteserverandinstallitthereusingthefoxx-managerutility.masterserversdonotknowwhichslavesareorwillbeconnectedtothem.Allserversinareplicationsetuparecurrentlyonlylooselycoupled.Therecurrentlyisnowayforaclienttoquerywhichserversarepresentinareplication.whennotusingourmesosintegrationfailovermustbehandledbyclientsorclientAPIs.therecurrentlyisonereplicationapplierperArangoDBdatabase.Itisthusnotpossibletohaveaslaveapplyoperationsfrommultiplemastersintothesametargetdatabase.replicationissetuponaper-databaselevel.WhenusingArangoDBwithmultipledatabases,replicationmustbeconfiguredindividuallyforeachdatabase.thereplicationapplierissingle-threaded,butwriteoperationsonthemastermaybeexecutedinparalleliftheyaffectdifferentcollections.Thusthereplicationappliermightnotbeabletocatchupwithaverypowerfulandloadedmaster.replicationisonlysupportedbetweenthetwoArangoDBserversrunningthesameArangoDBversion.ItiscurrentlynotpossibletoreplicatebetweendifferentArangoDBversions.areplicationappliercannotapplydatafromitself.


488

SynchronousReplicationAtitscoresynchronousreplicationwillreplicatewriteoperationstomultiplehosts.ThisfeatureisonlyavailablewhenoperatingArangoDBinacluster.Wheneveracoordinatorexecutesasychronouslyreplicatedwriteoperationitwillonlybereportedtobesuccessfulifitwascarriedoutonallreplicas.IncontrasttomultimasterreplicationsetupsknownfromothersystemsArangoDB'ssynchronousoperationguaranteesaconsistentstateacrossthecluster.


489

Implementation

Architectureinsidethecluster

SynchronousreplicationcanbeconfiguredpercollectionviathepropertyreplicationFactor.Synchronousreplicationrequiresaclustertooperate.

WheneveryouspecifyareplicationFactorgreaterthan1whencreatingacollection,synchronousreplicationwillbeactivatedforthiscollection.Theclusterwilldeterminesuitableleadersandfollowersforeveryrequestedshard(numberOfShards)withinthecluster.Whenrequestingdataofashardonlythecurrentleaderwillbeaskedwhereasfollowerswillonlykeeptheircopyinsync.Thisisduetothecurrentimplementationoftransactions.

Usingsynchronousreplicationalonewillguaranteeconsistencyandhighavailabiltyatthecostofreducedperformance:Writerequestswillhaveahigherlatency(duetoeverywrite-requesthavingtobeexecutedonthefollowers)andreadrequestswon'tscaleoutasonlytheleaderisbeingasked.

Inaclustersynchronousreplicationwillbemanagedbythecoordinatorsfortheclient.Thedatawillalwaysbestoredonprimaries.

ThefollowingexamplewillgiveyouanideaofhowsynchronousoperationhasbeenimplementedinArangoDB.

1. Connecttoacoordinatorviaarangosh2. Createacollection

127.0.0.1:8530@_system>db._create("test",{"replicationFactor":2})

3. thecoordinatorwillfigureoutaleaderand1followerandcreate1shard(asthisisthedefault)

4. Insertdata

127.0.0.1:8530@_system>db.test.insert({"replication":"ᚇ"})

5. Thecoordinatorwillwritethedatatotheleader,whichinturnwillreplicateittothefollower.

6. Onlywhenbothweresuccessfultheresultisreportedtobesuccessful

{

"_id":"test/7987",

"_key":"7987",

"_rev":"7987"

}

Whenafollowerfails,theleaderwillgiveuponitafter3secondsandproceedwiththeoperation.Assoonasthefollower(orthenetworkconnectiontotheleader)isbackup,thetwowillresynchronizeandsynchronousreplicationisresumed.Thishappensalltransparentlytotheclient.

ThecurrentimplementationofArangoDBdoesnotallowchangingthereplicationFactorlater.Thisissubjecttochange.Inthemeantimetheonlywayistodumpandrestorethecollection.Seethecookbookrecipeaboutmigrating.

Automaticfailover

Whenevertheleaderofashardisfailingandthereisaquerytryingtoaccessdataofthatshardthecoordinatorwillcontinuetryingtocontacttheleaderuntilittimeouts.Theinternalclustersupervisionrunningontheagencywillcheckclusterhealtheveryfewsecondsandwilltakeactionifthereisnoheartbeatfromaserverfor15seconds.Iftheleaderdoesn'tcomebackintimethesupervisionwillreorganizetheclusterbypromotingforeachshardafollowerthatisinsyncwithitsleadertobethenewleader.Fromthenonthecoordinatorswillcontactthenewleader.

Theprocessisbestoutlinedusinganexample:

1. Theleaderofashard(letsnameitDBServer001)isgoingdown.2. Acoordinatorisaskedtoreturnadocument:


490

127.0.0.1:8530@_system>db.test.document("100069")

3. ThecoordinatordetermineswhichserverisresponsibleforthisdocumentandfindsDBServer001

4. ThecoordinatortriestocontactDBServer001andtimeoutsbecauseitisnotreachable.5. Afterashortwhilethesupervision(runninginparallelontheagency)willseethatheartbeatsfromDBServer001arenotcomingin6. Thesupervisionpromotesoneofthefollowers(sayDBServer002)thatisinsynctobeleaderandmakesDBServer001afollower.7. AsthecoordinatorcontinuestryingtofetchthedocumentitwillseethattheleaderchangedtoDBServer0028. Thecoordinatortriestocontactthenewleader(DBServer002)andreturnstheresult:

{

"_key":"100069",

"_id":"test/100069",

"_rev":"513",

"replication":"ᚇ"

}

9. AfterawhilethesupervisiondeclaresDBServer001tobecompletelydead.10. AnewfollowerisdeterminedfromthepoolofDBservers.11. Thenewfollowersyncsitsdatafromtheleaderandorderisrestored.

Pleasenotethattheremaystillbetimeouts.Dependingonwhenexactlytherequesthasbeendone(inregardtothesupervision)anddependingonthetimeneededtoreconfiguretheclusterthecoordinatormightfailwithatimeouterror!


491

Configuration

Requirements

SynchronousreplicationrequiresanoperationalArangoDBcluster.

Enablingsynchronousreplication

Synchronousreplicationcanbeenabledpercollection.WhencreatingacollectionyoumayspecifythenumberofreplicasusingthereplicationFactorparameter.Thedefaultvalueissetto1whicheffectivelydisablessynchronousreplication.

Example:

127.0.0.1:8530@_system>db._create("test",{"replicationFactor":3})

Intheabovecase,anywriteoperationwillrequire2replicastoreportsuccessfromnowon.

Preparinggrowth

Youmaycreateacollectionwithhigherreplicationfactorthanavailable.Whenadditionaldbserversbecomeavailabletheshardsareautomaticallyreplicatedtothenewlyavailablemachines.

Multiplereplicasofthesameshardcannevercoexistonthesamedbserverinstance.


492

SatelliteCollectionsSatelliteCollectionsareanEnterpriseonlyfeature.WhendoingJoinsinanArangoDBclusterdatahastoexchangedbetweendifferentservers.

Joinswillbeexecutedonacoordinator.Itwillprepareanexecutionplanandexecuteit.Whenexecutingthecoordinatorwillcontactallshardsofthestartingpointofthejoinandaskfortheirdata.Thedatabaseserverscarryingoutthisoperationwillloadalltheirlocaldataandthenasktheclusterfortheotherpartofthejoin.Thisagainwillbedistributedtoallinvolvedshardsofthisjoinpart.

Insumthisresultsinmuchnetworktrafficandslowresultsdependingoftheamountofdatathathastobesentthroughoutthecluster.

Satellitecollectionsarecollectionsthatareintendedtoaddressthisissue.

Theywillfacilitatethesynchronousreplicationandreplicateallitsdatatoalldatabaseserversthatarepartofthecluster.

ThisenablesthedatabaseserverstoexecutethatpartofanyJoinlocally.

Thisgreatlyimprovesperformanceforsuchjoinsatthecostsofincreasedstoragerequirementsandpoorerwriteperformanceonthisdata.

TocreateasatellitecollectionsetthereplicationFactorofthiscollectionto"satellite".

Usingarangosh:

arangosh>db._create("satellite",{"replicationFactor":"satellite"});

Afullexample

arangosh>varexplain=require("@arangodb/aql/explainer").explain

arangosh>db._create("satellite",{"replicationFactor":"satellite"})

arangosh>db._create("nonsatellite",{numberOfShards:8})

arangosh>db._create("nonsatellite2",{numberOfShards:8})

Let'sanalyseanormaljoinnotinvolvingsatellitecollections:

arangosh>explain("FORdocinnonsatelliteFORdoc2innonsatellite2RETURN1")

Querystring:

FORdocinnonsatelliteFORdoc2innonsatellite2RETURN1

Executionplan:

IdNodeTypeSiteEst.Comment

1SingletonNodeDBS1*ROOT

4CalculationNodeDBS1-LET#2=1/*jsonexpression*//*constassignment*/

2EnumerateCollectionNodeDBS0-FORdocINnonsatellite/*fullcollectionscan*/

12RemoteNodeCOOR0-REMOTE

13GatherNodeCOOR0-GATHER

6ScatterNodeCOOR0-SCATTER

7RemoteNodeDBS0-REMOTE

3EnumerateCollectionNodeDBS0-FORdoc2INnonsatellite2/*fullcollectionscan*/



5ReturnNodeCOOR0-RETURN#2

Indexesused:

none

Optimizationrulesapplied:

IdRuleName

1move-calculations-up

2scatter-in-cluster

3remove-unnecessary-remote-scatter


493

Allshardsinvolvedqueryingthenonsatellitecollectionwillfanoutviathecoordinatortotheshardsofnonsatellite.Insum8shardswillopen8connectionstothecoordinatoraskingfortheresultsofthenonsatellite2join.Thecoordinatorwillfanouttothe8shardsofnonsatellite2.Sotherewillbequitesomenetworktraffic.

Let'snowhavealookatthesameusingsatellitecollections:

arangosh>db._query("FORdocinnonsatelliteFORdoc2insatelliteRETURN1")

Querystring:

FORdocinnonsatelliteFORdoc2insatelliteRETURN1

Executionplan:

IdNodeTypeSiteEst.Comment

1SingletonNodeDBS1*ROOT

4CalculationNodeDBS1-LET#2=1/*jsonexpression*//*constassignment*/

2EnumerateCollectionNodeDBS0-FORdocINnonsatellite/*fullcollectionscan*/

3EnumerateCollectionNodeDBS0-FORdoc2INsatellite/*fullcollectionscan,satellite*/



5ReturnNodeCOOR0-RETURN#2

Indexesused:

none

Optimizationrulesapplied:

IdRuleName

1move-calculations-up

2scatter-in-cluster

3remove-unnecessary-remote-scatter

4remove-satellite-joins

Inthisscenarioallshardsofnonsatellitewillbecontacted.Howeverasthejoinisasatellitejoinallshardscandothejoinlocallyasthedataisreplicatedtoallserversreducingthenetworkoverheaddramatically.

Caveats

Theclusterwillautomaticallykeepallsatellitecollectionsonallserversinsyncbyfacilitatingthesynchronousreplication.Thismeansthatwritewillbeexecutedontheleaderonlyandthisserverwillcoordinatereplicationtothefollowers.Ifafollowerdoesn'tanswerintime(duetonetworkproblems,temporaryshutdownetc.)itmayberemovedasafollower.ThisisbeingreportedtotheAgency.

Thefollower(oncebackinbusiness)willthenperiodicallychecktheAgencyandknowthatitisoutofsync.Itwillthenautomaticallycatchup.Thismaytakeawhiledependingonhowmuchdatahastobesynced.WhendoingajoininvolvingthesatelliteyoucanspecifyhowlongtheDBServerisallowedtowaitforsyncuntilthequeryisbeingaborted.

CheckAccessingCursorsfordetails.

DuringnetworkfailurethereisalsoaminimalchancethataquerywasproperlydistributedtotheDBServersbutthataprevioussatellitewritecouldnotbereplicatedtoafollowerandtheleaderdroppedthefollower.Thefollowerhoweveronlycheckseveryfewsecondsifitisreallyinsyncsoitmightindeeddeliverstaleresults.


494

ShardingArangoDBisorganizingitscollectiondatainshards.ShardingallowstousemultiplemachinestorunaclusterofArangoDBinstancesthattogetherconstituteasingledatabase.Thisenablesyoutostoremuchmoredata,sinceArangoDBdistributesthedataautomaticallytothedifferentservers.Inmanysituationsonecanalsoreapabenefitindatathroughput,againbecausetheloadcanbedistributedtomultiplemachines.

Shardsareconfiguredpercollectionsomultipleshardsofdataformthecollectionasawhole.TodetermineinwhichshardthedataistobestoredArangoDBperformsahashacrossthevalues.Bydefaultthishashisbeingcreatedfrom_key.

Toconfigurethenumberofshards:

127.0.0.1:8529@_system>db._create("sharded_collection",{"numberOfShards":4});

Toconfigurethehashingforanotherattribute:

127.0.0.1:8529@_system>db._create("sharded_collection",{"numberOfShards":4,"shardKeys":["country"]});

Thiswouldbeusefultokeepdataofeverycountryinoneshardwhichwouldresultinbetterperformanceforqueriesworkingonapercountrybase.YoucanalsospecifymultipleshardKeys.Notehoweverthatifyouchangetheshardkeysfromtheirdefault["_key"],thenfindingadocumentinthecollectionbyitsprimarykeyinvolvesarequesttoeverysingleshard.Furthermore,inthiscaseonecannolongerprescribetheprimarykeyvalueofanewdocumentbutmustusetheautomaticallygeneratedone.Thislatterrestrictioncomesfromthefactthatensuringuniquenessoftheprimarykeywouldbeveryinefficientiftheusercouldspecifytheprimarykey.

Onwhichnodeinaclusteraparticularshardiskeptisundefined.Thereisnooptiontoconfigureanaffinitybasedoncertainshardkeys.

Uniqueindexes(hash,skiplist,persistent)onshardedcollectionsareonlyallowedifthefieldsusedtodeterminetheshardkeyarealsoincludedinthelistofattributepathsfortheindex:

shardKeys indexKeys

a a ok

a b notok

a a,b ok

a,b a notok

a,b b notok

a,b a,b ok

a,b a,b,c ok

a,b,c a,b notok

a,b,c a,b,c ok

Sharding

495

GeneralUpgradeInformation

Recommendedmajorupgradeprocedure

ToupgradeanexistingArangoDB2.xto3.0pleaseusetheproceduredescribedhere.

Recommendedminorupgradeprocedure

ToupgradeanexistingArangoDBdatabasetoanewerversionofArangoDB(e.g.3.0to3.1,or3.1to3.2),thefollowingmethodisrecommended:

ChecktheCHANGELOGandthelistofincompatiblechangesforAPIorotherchangesinthenewversionofArangoDBandmakesureyourapplicationscandealwiththemStopthe"old"arangodserviceorbinaryCopytheentire"old"datadirectorytoasafeplace(thatis,abackup)InstallthenewversionofArangoDBandstarttheserverwiththe--database.auto-upgradeoptiononce.ThismightwritetothelogfileofArangoDB,soyoumaywanttocheckthelogsforanyissuesbeforegoingon.Startthe"new"arangodserviceorbinaryregularlyandcheckthelogsforanyissues.Whenyou'reconfidenteverythingwentwell,youmaywanttocheckthedatabasedirectoryforanyfileswiththeending.old.ThesefilesarecreatedbyArangoDBduringupgradesandcanbesafelyremovedmanuallylater.

Ifanythinggoeswrongduringorshortlyaftertheupgrade:

Stopthe"new"arangodserviceorbinaryReverttothe"old"arangodbinaryandrestorethe"old"datadirectoryStartthe"old"versionagain

ItisnotsupportedtousedatafilescreatedormodifiedbyanewerversionofArangoDBwithanolderArangoDBversion.Forexample,itisunsupportedandislikelytocauseproblemswhenusing3.2datafileswithanArangoDB3.0instance.

Switchingthestorageengine

Inordertouseadifferentstorageenginewithanexistingdatadirectory,itisrequiredtofirstcreatealogicalbackupofthedatausingarangodump.

Afterthat,thearangodservershouldberestartedwiththedesiredstorageengineselected(thiscanbedonebysettingtheoption--server.storage-engine)andusinganon-existingdatadirectory.

Whentheserverisupandrunningwiththedesiredstorageengine,thedatacanbere-importedusingarangorestore.

Upgrading

496

UpgradingtoArangoDB3.3PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB3.3.Pleasebesurethatyouhavecheckedthelistofchangesin3.3beforeupgrading.

Upgradingto3.3

497


Switchingthestorageengine

Inordertouseadifferentstorageenginewithanexistingdatadirectory,itisrequiredtofirstcreatealogicalbackupofthedatausingarangodump.Thatbackupshouldbecreatedbeforetheupgradeto3.2.

Afterthat,theArangoDBinstallationcanbeupgradedandstopped.Theservershouldthenberestartedwiththedesiredstorageengineselected(thiscanbedonebysettingtheoption--server.storage-engine)andusinganon-existingdatadirectory.Thiswillstarttheserverwiththeselectedstorageenginebutwithnodata.

Whentheserverisupandrunning,thedatafromthelogicalbackupcanbere-importedusingarangorestore.

Upgradingto3.2

498


Upgradingto3.1

499


MigratingdatabasesandcollectionsfromArangoDB2.8to3.0

ArangoDB3.0doesnotprovideanautomaticupdatemechanismfordatabasedirectoriescreatedwiththe2.xbranchesofArangoDB.

InordertomigratedatafromArangoDB2.8(oranolder2.xversion)intoArangoDB3.0,itisnecessarytoexportthedatafrom2.8usingarangodump,andthenimportthedumpintoafreshArangoDB3.0witharangorestore.

Todothis,firstrunthe2.8versionofarangodumptoexportthedatabasedataintoadirectory.arangodumpwilldumpthe_systemdatabasebydefault.Inordertomakeitdumpmultipledatabases,itneedstobeinvokedoncepersourcedatabase,e.g.

#in2.8

arangodump--server.database_system--output-directorydump-system

arangodump--server.databasemydb--output-directorydump-mydb

...

Thatwillproduceadumpdirectoryforeachdatabasethatarangodumpiscalledfor.Iftheserverhasauthenticationturnedon,itmaybenecessarytoprovidetherequiredcredentialswheninvokingarangodump,e.g.

arangodump--server.database_system--server.usernamemyuser--server.passwordmypasswd--output-directorydump-system

ThedumpsproducedbyarangodumpcannowbeimportedintoArangoDB3.0usingthe3.0versionofarangodump:

#in3.0

arangorestore--server.database_system--input-directorydump-system

arangorestore--server.databasemydb--input-directorydump-mydb

...

arangorestorewillbydefaultfailifthetargetdatabasedoesnotexist.Itcanbetoldtocreateitautomaticallyusingtheoption--create-databasetrue:

arangorestore--server.databasemydb--create-databasetrue--input-directorydump-mydb

Andagainitmayberequiredtoprovideaccesscredentialswheninvokingarangorestore:

arangorestore--server.databasemydb--create-databasetrue--server.usernamemyuser--server.passwordmypasswd--input-directo

rydump-system

Pleasenotethattheversionofdump/restoreshouldmatchtheserverversion,i.e.itisrequiredtodumptheoriginaldatawiththe2.8versionofarangodumpandrestoreitwiththe3.0versionofarangorestore.

Afterthatthe3.0instanceofArangoDBwillcontainthedatabasesandcollectionsthatwerepresentinthe2.8instance.

Adjustingauthenticationinfo

AuthenticationinformationwasstoredperdatabaseinArangoDB2.8,meaningtherecouldbedifferentusersandaccesscredentialsperdatabase.In3.0,theusersarestoredinacentrallocationinthe_systemdatabase.Tousethesameusersetupasin2.8,itmayberequiredtocreateextrausersand/oradjusttheirpermissions.

Inordertodothat,pleaseconnecttothe3.0instancewithanArangoShell(thiswillconnecttothe_systemdatabasebydefault):

arangosh--server.usernamemyuser--server.passwordmypasswd

Upgradingto3.0

500

Usethefollowingcommandstocreateanewuserwithsomepasswordandgrantthemaccesstoaspecificdatabase

require("@arangodb/users").save(username,password,true);

require("@arangodb/users").grantDatabase(username,databaseName,"rw");

Forexample,tocreateausermyuserwithpasswordmypasswdandgivethemaccesstodatabasesmydb1andmydb2,thecommandswouldlookasfollows:

require("@arangodb/users").save("myuser","mypasswd",true);

require("@arangodb/users").grantDatabase("myuser","mydb1","rw");

require("@arangodb/users").grantDatabase("myuser","mydb2","rw");

Existinguserscanalsobeupdated,removedorlistedusingthefollowingcommands:

/*updateusermyuserwithpasswordmypasswd*/

require("@arangodb/users").update("myuser","mypasswd",true);

/*removeusermyuser*/

require("@arangodb/users").remove("myuser");

/*listallusers*/

require("@arangodb/users").all();

Foxxapplications

Thedump/restoreproceduredescribedabovewillnotexportandre-importFoxxapplications.Inordertomovethesefrom2.8to3.0,Foxxapplicationsshouldbeexportedaszipfilesviathe2.8webinterface.

Thezipfilescanthenbeuploadedinthe"Services"sectionintheArangoDB3.0webinterface.Applicationsmayneedtobeadjustedmanuallytorunin3.0.PleaseconsultthemigrationguideforFoxxapps.

AnalternativewayofmovingFoxxappsinto3.0istocopythesourcedirectoryofa2.8Foxxapplicationmanuallyintothe3.0Foxxappsdirectoryforthetargetdatabase(whichisnormally/var/lib/arangodb3-apps/_db/<dbname>/buttheexactlocationisplatform-specific).

Upgradingto3.0

501


PleasenotefirstthatadatabasedirectoryusedwithArangoDB2.8cannotbeusedwithearlierversions(e.g.ArangoDB2.7)anymore.Upgradingadatabasedirectorycannotbereverted.ThereforepleasemakesuretocreateafullbackupofyourexistingArangoDBinstallationbeforeperforminganupgrade.

DatabaseDirectoryVersionCheckandUpgrade

ArangoDBwillperformadatabaseversioncheckatstartup.WhenArangoDB2.8encountersadatabasecreatedwithearlierversionsofArangoDB,itwillrefusetostart.Thisisintentional.

Theoutputwillthenlooklikethis:

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Databasedirectoryversion(20702)islowerthancurrentversion(208

00).

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':---------------------------------------------------------------------

-

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':ItseemslikeyouhaveupgradedtheArangoDBbinary.

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Ifthisiswhatyouwantedtodo,pleaserestartwiththe

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':--upgrade

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':optiontoupgradethedatainthedatabasedirectory.

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':Normallyyoucanusethecontrolscripttoupgradeyourdatabase

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbstop

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbupgrade

2015-12-04T17:11:17Z[31432]ERRORIndatabase'_system':/etc/init.d/arangodbstart


-

2015-12-04T17:11:17Z[31432]FATALDatabase'_system'needsupgrade.Pleasestarttheserverwiththe--upgradeoption

TomakeArangoDB2.8startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.

Note:herethesamedatabaseshouldbespecifiedthatisalsospecifiedwhenarangodisstartedregularly.Pleasedonotrunthe--upgradecommandoneachindividualdatabasesubfolder(nameddatabase-<somenumber>).

Forexample,ifyouregularlystartyourArangoDBserverwith

unix>arangodmydatabasefolder

thenrunning

unix>arangodmydatabasefolder--upgrade

willperformtheupgradeforthewholeArangoDBinstance,includingallofitsdatabases.

Startingwith--upgradewillrunadatabaseversioncheckandperformanynecessarymigrations.Asusual,youshouldcreateabackupofyourdatabasedirectorybeforeperformingtheupgrade.

Thelastlineoftheoutputshouldlooklikethis:

2015-12-04T17:12:15Z[31558]INFOdatabaseupgradepassed

Pleasecheckthefulloutputthe--upgraderun.Upgradingmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolvedmanually,youcanstartArangoDB2.8regularly.

Upgradingaclusterplannedinthewebinterface

Upgradingto2.8

502

AclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.

Wehavetriedtomakethisprocedureaspainlessandconvenientforyou.Weassumethatyouplanned,launchedandadministratedaclusterusingthegraphicalfrontendinyourbrowser.Theupgradeprocedureisthenasfollows:

1. Firstshutdownyourclusterusingthegraphicalfrontendasusual.

2. Thenupgradealldispatcherinstancesonallmachinesinyourclusterusingtheversioncheckasdescribedaboveandrestartthem.

3. Nowopentheclusterdashboardinyourbrowserbypointingittothesamedispatcherthatyouusedtoplanandlaunchtheclusterinthegraphicalfrontend.Inadditiontotheusualbuttons"Relaunch","Editclusterplan"and"Deleteclusterplan"youwillseeanotherbuttonmarked"Upgradeandrelaunchcluster".

4. Hitthisbutton,yourclusterwillbeupgradedandlaunchedandallisdoneforyoubehindthescenes.Ifallgoeswell,youwillseetheusualclusterdashboardafterafewseconds.Ifthereisanerror,youhavetoinspectthelogfilesofyourclusterArangoDBinstances.Pleaseletusknowifyourunintoproblems.

ThereisanalternativewayusingtheArangoDBshell.Insteadofsteps3.and4.aboveyoucanlauncharangosh,pointittothedispatcherthatyouhaveusedtoplanandlaunchtheclusterusingtheoption--server.endpoint,andexecute

arangosh>require("org/arangodb/cluster").Upgrade("root","");

Thisupgradestheclusterandlaunchesit,exactlyaswiththebuttonaboveinthegraphicalfrontend.Youhavetoreplace"root"withausernameand""withapasswordthatisvalidforauthenticationwiththecluster.

UpgradingFoxxappsgeneratedbyArangoDB2.7andearlier

TheimplementationoftherequirefunctionusedtoimportmodulesinArangoDBandFoxxhaschangedinordertoimprovecompatibilitywithNode.jsmodules.

Givenanapp/servicewiththefollowinglayout:

manifest.jsoncontrollers/

todos.jsmodels/

todo.jsrepositories/

todos.jsnode_modules/

models/todo.js

Thefilecontrollers/todos.jswouldpreviouslycontainthefollowingrequirecalls:

var_=require('underscore');

varjoi=require('joi');

varFoxx=require('org/arangodb/foxx');

varArangoError=require('org/arangodb').ArangoError;

varTodos=require('repositories/todos');//<--!

varTodo=require('models/todo');//<--!

Therequirepathsrepositories/todosandmodels/todowerepreviouslyresolvedlocallyasrelativetotheapproot.

Startingwith2.8thesepathswouldinsteadberesolvedasrelativetothenode_modulesfolderortheglobalArangoDBmodulepathsbeforebeingresolvedlocallyasafallback.

Inthegivenexamplelayouttheappwouldbreakin2.8becausethemodulenamemodels/todowouldalwaysresolvetonode_modules/models/todo.js(whichpreviouslywouldhavebeenignored)insteadofthelocalmodels/todo.js.

Inordertomakesuretheappstillworksin2.8,therequirecallsincontrollers/todos.jswouldneedtobeadjustedtolooklikethis:

Upgradingto2.8

503

var_=require('underscore');

varjoi=require('joi');


varArangoError=require('org/arangodb').ArangoError;

varTodos=require('../repositories/todos');//<--!

varTodo=require('../models/todo');//<--!

Notethattheold"global"stylerequirecallsmaystillworkin2.8butmaybreakunexpectedlyifmoduleswithmatchingnamesareinstalledglobally.

Upgradingto2.8

504







0).

2015-02-17T09:43:11Z[8302]ERRORIndatabase'_system':----------------------------------------------------------------------











TomakeArangoDB2.6startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.




thenrunning








Upgradingto2.6

505










Upgradingto2.6

506



In2.5wehavealsochangedthepathsforFoxxapplications.PleasealsomakesurethatyouhaveabackupofallFoxxappsinyourjavascript.app-pathandjavascript.dev-app-path.ItissufficienttohavethesourcefilesforFoxxsomewhereelsesoyoucanreinstallthemonerror.Tocheckthateverythinghasworkedduringupgradeyoucouldusetheweb-interfaceApplicationstabor

unix>foxx-managerlist

forallyourdatabases.Thelistedappsshouldbeidenticalbeforeandaftertheupgrade.





0).












TomakeArangoDB2.5startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption.Note:WehavechangedFoxxfolderstructureandimplementedanupgradetasktomoveyourapplicationstothenewstructure.InordertotellthisupgradetasktoalsomoveyourdevelopmentFoxxappspleasemakesureyougivethedev-app-pathaswell.IfyouhavenotuseddevelopmentmodeforFoxxappsyoucandropthe--javascript.dev-app-path.Itisonlypossibletoupgradeonedev-app-pathtogetherwithonedatafolder.

unix>arangoddata--upgrade--javascript.dev-app-pathdevapps

wheredataisArangoDB'smaindatadirectoryanddevappsisthedirectorywhereyoudevelopFoxxapps.




thenrunning


Upgradingto2.5

507






UpgradingaclusterplannedinthewebinterfaceAclusterofArangoDBinstanceshastobeupgradedaswell.ThisinvolvesupgradingallArangoDBinstancesinthecluster,aswellasrunningtheversioncheckonthewholerunningclusterintheend.









Upgradingto2.5

508







00).


-










-


TomakeArangoDB2.4startwithadatabasedirectorycreatedwithanearlierArangoDBversion,youmayneedtoinvoketheupgradeprocedureonce.ThiscanbedonebyrunningArangoDBfromthecommandlineandsupplyingthe--upgradeoption:

unix>arangoddata--upgrade

wheredataisArangoDB'smaindatadirectory.




thenrunning






Upgradingto2.4

509












Upgradingto2.4

510






2014-11-03T15:48:06Z[2694]ERRORIndatabase'_system':Databasedirectoryversion(2.2)islowerthancurrentversion(20300)

.


















thenrunning




Theoutputshouldlooklikethis:

2014-11-03T15:48:47Z[2708]INFOIndatabase'_system':Found24definedtask(s),5task(s)torun

2014-11-03T15:48:47Z[2708]INFOIndatabase'_system':stateprod/standalone/upgrade,tasksupdateUserModel,createStatistics,

upgradeClusterPlan,setupQueues,setupJobs

2014-11-03T15:48:48Z[2708]INFOIndatabase'_system':upgradesuccessfullyfinished


Upgradingto2.3

511

Pleasechecktheoutputthe--upgraderun.Itmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolved,youcanstartArangoDB2.3regularly.











Upgradingto2.3

512

UpgradingtoArangoDB2.2PleasereadthefollowingsectionsifyouupgradefromapreviousversiontoArangoDB2.2.





2014-07-07T22:04:53Z[18675]ERRORIndatabase'_system':Databasedirectoryversion(2.1)islowerthanserverversion(2.2).


-










-

2014-07-07T22:04:53Z[18675]FATALDatabaseversioncheckfailedfor'_system'.Pleasestarttheserverwiththe--upgradeopti

on







thenrunning




Theoutputshouldlooklikethis:

2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':startingupgradefromversion2.1to2.2.0

2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':Found19definedtask(s),2task(s)torun

2014-07-07T22:11:30Z[18867]INFOIndatabase'_system':upgradesuccessfullyfinished


Upgradingto2.2

513

Pleasechecktheoutputthe--upgraderun.Itmayproduceerrors,whichneedtobefixedbeforeArangoDBcanbeusedproperly.Ifnoerrorsarepresentortheyhavebeenresolved,youcanstartArangoDB2.2regularly.











Upgradingto2.2

514

Troubleshooting

Troubleshooting

515

ArangodIftheArangoDBserverdoesnotstartorifyoucannotconnecttoitusingarangoshorotherclients,youcantrytofindtheproblemcausebyexecutingthefollowingsteps.Iftheserverstartsupwithoutproblemsyoucanskipthissection.

Checktheserverlogfile:Iftheserverhaswrittenalogfileyoushouldcheckitbecauseitmightcontainrelevanterrorcontextinformation.

Checktheconfiguration:Theserverlooksforaconfigurationfilenamedarangod.confonstartup.Thecontentsofthisfilewillbeusedasabaseconfigurationthatcanoptionallybeoverriddenwithcommand-lineconfigurationparameters.Youshouldchecktheconfigfileforthemostrelevantparameterssuchas:

server.endpoint:WhatIPaddressandporttobindtologparameters:Ifandwheretologdatabase.directory:Paththedatabasefilesarestoredin

Iftheconfigurationrevealsthatsomethingisnotconfiguredrighttheconfigfileshouldbeadjustedandtheserverberestarted.

Starttheservermanuallyandcheckitsoutput:Startingtheservermightfailevenbeforeloggingisactivatedsotheserverwillnotproducelogoutput.Thiscanhappeniftheserverisconfiguredtowritethelogstoafilethattheserverhasnopermissionson.Inthiscasetheservercannotloganerrortothespecifiedlogfilebutwillwriteastartuperroronstderrinstead.Startingtheservermanuallywillalsoallowyoutooverridespecificconfigurationoptions,e.g.toturnon/offfileorscreenloggingetc.

ChecktheTCPport:Iftheserverstartsupbutdoesnotacceptanyincomingconnectionsthismightbeduetofirewallconfigurationbetweentheserverandanyclient(s).TheserverbydefaultwilllistenonTCPport8529.PleasemakesurethisportisactuallyaccessiblebyotherclientsifyouplantouseArangoDBinanetworksetup.

Whenusinghostnamesintheconfigurationorwhenconnecting,pleasemakesurethehostnameisactuallyresolvable.ResolvinghostnamesmightinvokeDNS,whichcanbeasourceoferrorsonitsown.

ItisgenerallygoodadvicetonotuseDNSwhenspecifyingtheendpointsandconnectionaddresses.UsingIPaddressesinsteadwillruleoutDNSasasourceoferrors.Anotheralternativeistouseahostnamespecifiedinthelocal/etc/hostsfile,whichwillthenbypassDNS.

Testifcurlcanconnect:Oncetheserverisstarted,youcanquicklyverifyifitrespondstorequestsatall.Thischeckallowsyoutodeterminewhetherconnectionerrorsareclient-specificornot.Ifatleastoneclientcanconnect,itislikelythatconnectionproblemsofotherclientsarenotduetoArangoDB'sconfigurationbutduetoclientorin-betweennetworkconfigurations.

Youcantestconnectivityusingasimplecommandsuchas:

curl--dump--XGEThttp://127.0.0.1:8529/_api/version&&echo

ThisshouldreturnaresponsewithanHTTP200statuscodewhentheserverisrunning.Ifitdoesitalsomeanstheserverisgenerallyacceptingconnections.Alternativetoolstocheckconnectivityarelynxorab.

arangod

516

http://127.0.0.1:8529/_api/version

EmergencyConsoleTheArangoDBdatabaseserverhastwomodesofoperation:Asaserver,whereitwillanswertoclientrequestsandasanemergencyconsole,inwhichyoucanaccessthedatabasedirectly.Thelatter-asthenamesuggests-shouldonlybeusedincaseofanemergency,forexample,acorruptedcollection.Usingtheemergencyconsoleallowsyoutoissueallcommandsnormallyavailableinactionsandtransactions.Whenstartingtheserverinemergencyconsolemode,theservercannothandleanyclientrequests.

Youshouldneverstartmorethanoneserverusingthesamedatabasedirectory,independentofthemodeofoperation.Normally,ArangoDBwillpreventyoufromdoingthisbyplacingalockfileinthedatabasedirectoryandnotallowingasecondArangoDBinstancetousethesamedatabasedirectoryifalockfileisalreadypresent.

InCaseOfDisaster

Thefollowingcommandstartsanemergencyconsole.

Note:Neverstarttheemergencyconsoleforadatabasewhichalsohasaserverattachedtoit.Ingeneral,theArangoDBshelliswhatyouwant.

>./arangod--console--logerror/tmp/vocbase

ArangoDBshell[V8version5.0.71.39,DBversion3.x.x]

arango>1+2;

3

arango>vardb=require("@arangodb").db;db.geo.count();

703

TheemergencyconsoleprovidesaJavaScriptconsoledirectlyrunninginthearangodserverprocess.ThisallowstodebugandexaminecollectionsanddocumentsaswiththenormalArangoDBshell,butwithoutclient/servercommunication.

However,itisverylikelythatyouwillneverneedtheemergencyconsoleunlessyouareanArangoDBdeveloper.

EmergencyConsole

517

DatafileDebugger

InCaseOfDisaster

AranagoDBusesappend-onlyjournals.Datacorruptionshouldonlyoccurwhenthedatabaseserveriskilled.Inthiscase,thecorruptionshouldonlyoccurinthelastobject(s)thathavebeingwrittentothejournal.

Ifacorruptionoccurswithinanormaldatafile,thenthiscanonlyhappenifahardwarefaultoccurred.

Ifajournalordatafileiscorrupt,shutdownthedatabaseserverandstarttheprogram

unix>arango-dfdb

inordertochecktheconsistencyofthedatafilesandjournals.Thisbringsup

_________________

/\___||____/_(_)|___/\/__\/_\

//\/_`|__/_`||_|||/_\//\/__\////_\/

//_//(_||||(_||_|||__///_//\/\//_\\

/___,'\__,_|\__\__,_|_||_|_|\___|/___,'\_____/\____/

Availablecollections:

0:_structures

1:_users

2:_routing

3:_modules

4:_graphs

5:products

6:prices

*:all

Collectiontocheck:

Youcannowselectwhichdatabaseandcollectionyouwanttocheck.Afteryouselectedoneorallofthecollections,aconsistencycheckwillbeperformed.

Checkingcollection#1:_users

Database

path:/usr/local/var/lib/arangodb

Collection

name:_users

identifier:82343

Datafiles

#ofjournals:1

#ofcompactors:1

#ofdatafiles:0

Datafile

path:/usr/local/var/lib/arangodb/collection-82343/journal-1065383.db

type:journal

currentsize:33554432

maximalsize:33554432

totalused:256

#ofentries:3

status:OK

Ifthereisaproblemwithoneofthedatafiles,thenthedatabasedebuggerwillprintitandpromptforwhethertoattempttofixit.

WARNING:Thejournalwasnotclosedproperly,thelastentriesarecorrupted.

ThismighthappenArangoDBwaskilledandthelastentrieswerenot

fullywrittentodisk.

DatafileDebugger

518

Wipethelastentries(Y/N)?

IfyouanswerY,thecorruptedentrywillberemoved.

Ifyouseeacorruptioninadatafile(andnotajournal),thensomethingisterriblywrong.ThesefilesareimmutableandneverchangedbyArangoDB.Acorruptioninsuchfileisanindicationofahard-diskfailure.

DatafileDebugger

519

ArangobenchArangobenchisArangoDB'sbenchmarkandtesttool.Itcanbeusedtoissuetestrequeststothedatabaseforperformanceandserverfunctiontesting.Itsupportsparallelqueryingandbatchrequests.

Relatedblogposts:

MeasuringArangoDBinsertperformanceGainfactorof5usingbatchrequests

Startupoptions--async:Sendasynchronousrequests.Thedefaultvalueisfalse.

--batch-size:Numberofoperationstosendperbatch.Use0todisablebatching(thisisthedefault).

--collection:Nameofcollectiontouseintest(onlyrelevantforteststhatinvokecollections).

--replication-factor:Incaseofacluster,thereplicationfactorofthecreatedcollections.

--number-of-shards:Incaseofacluster,thenumberofshardsofthecreatedcollections.

--wait-for-sync:ThevalueofwaitForSyncforcreatedcollections.

--complexity:Complexityvaluefortestcase(default:1).Meaningdependsontestcase.

--concurrency:Numberofparallelthreadsthatwillissuerequests(default:1).

--configuration:Readconfigurationfromfile.

--delay:Useastartupdelay.Thisisonlynecessarywhenruninseries.Thedefaultvalueisfalse.

--keep-alive:UseHTTPkeep-alive(default:true).

--progress:Showprogressofbenchmark,onevery20threquest.Settofalsetodisableintermediatelogging.Thedefaultvalueistrue.

--requests:Totalnumberofrequeststoperform(default:1000).

--server.endpoint:Serverendpointtoconnectto,consistingofprotocol,IPaddressandport.Defaultstotcp://localhost:8529.

--server.database:Databasenametousewhenconnecting(default:"_system").

--server.username:Usernametousewhenconnecting(default:"root").

--server.password:Passwordtousewhenconnecting.Don'tspecifythisoptiontogetapasswordprompt.

--server.authentication:Wetherornottoshowthepasswordpromptanduseauthenticationwhenconnectingtotheserver(default:true).

--test-case:Nameoftestcasetoperform(default:"version").Possiblevalues:

version:requests/_api/versiondocument:createsdocumentscollection:createscollectionsimport-document:createsdocumentsviatheimportAPIhash:Create/Read/Update/Readdocumentsindexedbyahashindexskiplist:Create/Read/Update/Readdocumentsindexedbyaskiplistedge:Create/Read/Updateedgedocumentsshapes:Create&Deletedocumentswithheterogeneousattributenamesshapes-append:Createdocumentswithheterogeneousattributenamesrandom-shapes:Create/Read/Deleteheterogeneousdocumentswithrandomvaluescrud:Create/Read/Update/Deletecrud-append:Create/Read/Update/Readagain

Arangobench

520

https://www.arangodb.com/2013/11/measuring-arangodb-insert-performance/

https://www.arangodb.com/2012/10/gain-factor-of-5-using-batch-updates/

crud-write-read:Create/ReadDocumentsaqltrx:AQLTransactionswithdeepnestedAQLFOR-loopscounttrx:usesJStransactionstocountthedocumentsandinserttheresultagainmultitrx:multipletransactionscombiningreads&writesfromjsmulti-collection:multipletransactionscombiningreads&writesfromjsonmultiplecollectionsaqlinsert:insertdocumentsviaAQLaqlv8:executeAQLwithV8functionstoinsertrandomdocuments

--verbose:PrintoutrepliesiftheHTTPheaderindicatesDBerrors.(default:false).

Examples

arangobench

StartsArangobenchwiththedefaultuserandserverendpoint.

--test-caseversion--requests1000--concurrency1

Runsthe'version'testcasewith1000requests,withoutconcurrency.

--test-casedocument--requests1000--concurrency2

Runsthe'document'testcasewith2000requests,withtwoconcurrentthreads.

--test-casedocument--requests1000--concurrency2--asynctrue

Runsthe'document'testcasewith2000requests,withconcurrency2,withasyncrequests.

--test-casedocument--requests1000--concurrency2--batch-size10

Runsthe'document'testcasewith2000requests,withconcurrency2,usingbatchrequests.

Arangobench

521

Architecture

AppendOnly/MVCC

Insteadofoverwritingexistingdocuments,ArangoDBwillcreateanewversionofmodifieddocuments.Thisiseventhecasewhenadocumentgetsdeleted.Thetwobenefitsare:

Objectscanbestoredcoherentlyandcompactlyinthemainmemory.Objectsarepreserved,isolatedwritingandreadingtransactionsallowaccessingtheseobjectsforparalleloperations.

Thesystemcollectsobsoleteversionsasgarbage,recognizingthemasforsaken.Garbagecollectionisasynchronousandrunsparalleltootherprocesses.

MostlyMemory/Durability

Databasedocumentsarestoredinmemory-mappedfiles.Perdefault,thesememory-mappedfilesaresyncedregularlybutnotinstantly.Thisisoftenagoodtradeoffbetweenstorageperformanceanddurability.Ifthislevelofdurabilityistoolowforanapplication,theservercanalsosyncallmodificationstodiskinstantly.ThiswillgivefulldurabilitybutwillcomewithaperformancepenaltyaseachdatamodificationwilltriggerasyncI/Ooperation.

Architecture

522

Write-aheadlogBothstorageenginesuseaformofwriteaheadlogging(WAL).Startingwithversion2.2ArangoDBstoresalldata-modificationoperationinitswrite-aheadlog.Thewrite-aheadlogissequenceofappend-onlyfilescontainingallthewriteoperationsthatwereexecutedontheserver.

Itisusedtorundatarecoveryafteraservercrash,andcanalsobeusedinareplicationsetupwhenslavesneedtoreplaythesamesequenceofoperationsasonthemaster.

MMFilesWALDetails

Bydefault,eachwrite-aheadlogfileis32MiBinsize.Thissizeisconfigurableviatheoption--wal.logfile-size.Whenawrite-aheadlogfileisfull,itissettoread-only,andfollowingoperationswillbewrittenintothenextwrite-aheadlogfile.Bydefault,ArangoDBwillreservesomesparelogfilesinthebackgroundsoswitchinglogfilesshouldbefast.HowmanyreservelogfilesArangoDBwilltrytokeepavailableinthebackgroundcanbecontrolledbytheconfigurationoption--wal.reserve-logfiles.

Datacontainedinfullwrite-aheadfileswilleventuallybetransferredintothejournalsordatafilesofcollections.Onlythe"surviving"documentswillbecopiedover.Whenallremainingoperationsfromawrite-aheadlogfilehavebeencopiedoverintothejournalsordatafilesofthecollections,thewrite-aheadlogfilecansafelyberemovedifitisnotusedforreplication.

Long-runningtransactionspreventwrite-aheadlogfilesfrombeingfullygarbage-collectedbecauseitisunclearwhetheratransactionwillcommitorabort.Long-runningtransactionscanthusblockthegarbage-collectionprogressandshouldthereforebeavoidedatallcosts.

Onasystemthatactsasareplicationmaster,itisusefultokeepafewofthealreadycollectedwrite-aheadlogfilessoreplicationslavesstillcanfetchdatafromthemifrequired.Howmanycollectedlogfileswillbekeptbeforetheygetdeletedisconfigurableviatheoption--wal.historic-logfiles.

Forallwrite-aheadlogconfigurationoptions,pleaserefertothepageWrite-aheadlogoptions.

RocksDBWALDetails

TheoptionsmentionedaboveonlyapplyforMMFiles.TheWALintherocksdbstorageengineworksslightlydifferently.Note:InrocksdbtheWALoptionsareallprefixedwith--rocksdb.*.The--wal.*optionsdohavenoeffect.

TheindividualRocksDBWALfilesareperdefaultabout64MiBbig.Thesizewillalwaysbeproportionallysizedtothevaluespecifiedvia--rocksdb.write-buffer-size.Thevaluespecifiestheamountofdatatobuildupinmemory(backedbytheunsortedWALondisk)beforeconvertingittoasortedon-diskfile.Largervaluescanincreaseperformance,especiallyduringbulkloads.Upto--rocksdb.max-write-buffer-numberwritebuffersmaybeheldinmemoryatthesametime,soyoumaywishtoadjustthisparametertocontrolmemoryusage.Alargerwritebufferwillresultinalongerrecoverytimethenexttimethedatabaseisopened.

TheRocksDBWALonlycontainscommittedtransactions.Thismeansyouwillneverseepartialtransactionsinthereplicationlog,butitalsomeanstransactionsaretrackedcompletelyin-memory.InpracticethiscausesRocksDBtransactionsizestobelimited,formoreinformationseetheRocksDBConfiguration

Write-aheadlog

523

StorageEnginesAttheverybottomoftheArangoDBdatabaseliesthestorageengine.Thestorageengineisresponsibleforpersistingthedocumentsondisk,holdingcopiesinmemory,providingindexesandcachestospeedupqueries.

Uptoversion3.1ArangoDBonlysupportedmemorymappedfiles(MMFiles)assolestorageengine.Beginningwith3.2ArangoDBhassupportforpluggablestorageengines.ThesecondsupportedengineisRocksDBfromFacebook.

MMFiles RocksDB

default optional

datasetneedstofitintomemory workwithasmuchdataasfitsondisk

indexesinmemory hotsetinmemory,dataandindexesondisk

slowrestartduetoindexrebuilding faststartup(norebuildingofindexes)

volatilecollections(onlyinmemory,optional) collectiondataalwayspersisted

collectionlevellocking(writesblockreads) concurrentreadsandwrites

Blogarticle:ComparingnewRocksDBandMMFilesstorageengines

RocksDBisanembeddablepersistentkey-valuestore.Itisalogstructuredatabaseandisoptimizedforfaststorage.

TheMMFilesengineisoptimizedfortheuse-casewherethedatafitsintothemainmemory.Itallowsforveryfastconcurrentreads.However,writesblockreadsandlockingisoncollectionlevel.Indexesarealwaysinmemoryandarerebuiltonstartup.Thisgivesbetterperformancebutimposesalongerstartuptime.

TheRocksDBengineisoptimizedforlargedata-setsandallowsforasteadyinsertperformanceevenifthedata-setismuchlargerthanthemainmemory.Indexesarealwaysstoredondiskbutcachesareusedtospeedupperformance.RocksDBusesdocument-levellocksallowingforconcurrentwrites.Writesdonotblockreads.Readsdonotblockwrites.

Theenginemustbeselectedforthewholeserver/cluster.Itisnotpossibletomixengines.Thetransactionhandlingandwrite-ahead-logformatintheindividualenginesisverydifferentandthereforecannotbemixed.

RocksDB

Advantages

RocksDBisaveryflexibleenginethatcanbeconfiguredforvarioususecases.

ThemainadvantagesofRocksDBare

document-levellockssupportforlargedata-setspersistentindexes

Caveats

RocksDBallowsconcurrentwrites.However,whentouchingthesamedocumentawriteconflictisraised.ThiscannothappenwiththeMMFilesengine,thereforeapplicationsthatswitchtoRocksDBneedtobepreparedthatsuchexceptioncanarise.ItispossibletoexclusivelylockcollectionswhenexecutingAQL.Thiswillavoidwriteconflictsbutalsoinhibitsconcurrentwrites.

Currently,anotherrestrictionisduetothetransactionhandlinginRocksDB.Transactionsarelimitedintotalsize.Ifyouhaveastatementmodifyingalotofdocumentsitisnecessarytocommitdatainbetween.ThiswillbedoneautomaticallyforAQLbydefault.

Performance

RocksDBisabasedonlog-structuredmergetree.Agoodintroductioncanbefoundin:

StorageEngines

524

https://www.arangodb.com/why-arangodb/comparing-rocksdb-mmfiles-storage-engines/

http://www.benstopford.com/2015/02/14/log-structured-merge-trees/https://blog.acolyer.org/2014/11/26/the-log-structured-merge-tree-lsm-tree/

Thebasicideaisthatdataisorganizedinlevelswereeachlevelisafactorlargerthantheprevious.Newdatawillresideinsmallerlevelswhileolddataismoveddowntothelargerlevels.Thisallowstosupporthighrateofinsertsoveranextendedperiod.Inprincipleitispossiblethatthedifferentlevelsresideondifferentstoragemedia.ThesmalleronesonfastSSD,thelargeronesonbiggerspinningdisks.

RocksDBitselfprovidesalotofdifferentknobstofinetunethestorageengineaccordingtoyouruse-case.ArangoDBsupportsthemostcommononesusingtheoptionsbelow.

Performancereportsforthestorageenginecanbefoundhere:

https://github.com/facebook/rocksdb/wiki/performance-benchmarkshttps://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

ArangoDBoptions

ArangoDBhasacacheforthepersistentindexesinRocksDB.Thetotalsizeofthiscacheiscontrolledbytheoption

--cache.size

RocksDBalsohasacachefortheblocksstoredondisk.Thesizeofthiscacheiscontrolledbytheoption

--rocksdb.block-cache-size

ArangoDBdistributestheavailablememoryequallybetweenthetwocachesbydefault.

ArangoDBchoosesasizeforthevariouslevelsinRocksDBthatissuitableforgeneralpurposeapplications.

RocksDBlogstrutureddatalevelshaveincreasingsize

MEM:--

L0:--

L1:----

L2:--------

...

NeworupdatedDocumentsarefirststoredinmemory.Ifthismemtablereachesthelimitgivenby

--rocksdb.write-buffer-size

itwillconvertedtoanSSTfileandinsertedatlevel0.

Thefollowingoptioncontrolsthesizeofeachlevelandthedepth.

--rocksdb.num-levelsN

LimitsthenumberoflevelstoN.Bydefaultitis7andthereisseldomareasontochangethis.Anewlevelisonlyopenedifthereistoomuchdatainthepreviousone.

--rocksdb.max-bytes-for-level-baseB

L0willholdatmostBbytes.

--rocksdb.max-bytes-for-level-multiplierM

EachlevelisatmostMtimesasmuchbytesasthepreviousone.ThereforethemaximumnumberofbytesforlevelLcanbecalculatedas

max-bytes-for-level-base*(max-bytes-for-level-multiplier^(L-1))

StorageEngines

525

http://www.benstopford.com/2015/02/14/log-structured-merge-trees/

https://blog.acolyer.org/2014/11/26/the-log-structured-merge-tree-lsm-tree/

https://github.com/facebook/rocksdb/wiki/performance-benchmarks

https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

Future

RocksDBimposesalimitonthetransactionsize.Itisoptimizedtohandlesmalltransactionsveryefficiently,butiseffectivelylimitingthetotalsizeoftransactions.

ArangoDBcurrentlyusesRocksDB'stransactionstoimplementtheArangoDBtransactionhandling.ThereforethesamerestrictionsapplyforArangoDBtransactionswhenusingtheRocksDBengine.

WewillimprovethisbyintroducingdistributedtransactionsinafutureversionofArangoDB.ThiswillallowhandlinglargetransactionsasaseriesofsmallRocksDBtransactionsandhenceremovingthesizerestriction.

StorageEngines

526

ReleaseNotes

WhatsNew

WhatsNewin3.3WhatsNewin3.2WhatsNewin3.1WhatsNewin3.0WhatsNewin2.8WhatsNewin2.7WhatsNewin2.6WhatsNewin2.5WhatsNewin2.4WhatsNewin2.3WhatsNewin2.2WhatsNewin2.1

KnownIssues

KnownIssuesin3.2

Incompatiblechanges

AlsoseeUpgradingintheAdministrationchapter.

Incompatiblechangesin3.3Incompatiblechangesin3.2Incompatiblechangesin3.1Incompatiblechangesin3.0Incompatiblechangesin2.8Incompatiblechangesin2.7Incompatiblechangesin2.6Incompatiblechangesin2.5Incompatiblechangesin2.4Incompatiblechangesin2.3

Releasenotes

527

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB3.3.ArangoDB3.3alsocontainsseveralbugfixesthatarenotlistedhere.

Datacenter-to-datacenterreplication(DC2DC)

Everycompanyneedsadisasterrecoveryplanforallimportantsystems.Thisistruefromsmallunitslikesingleprocessesrunninginsomecontainertothelargestdistributedarchitectures.Fordatabasesinparticularthisusuallyinvolvesamixtureoffault-tolerance,redundancy,regularbackupsandemergencyplans.Thelargeradatastore,themoredifficultitistocomeupwithagoodstrategy.

Therefore,itisdesirabletobeabletorunadistributeddatabaseinonedata-centerandreplicatealltransactionstoanotherdata-centerinsomeway.Often,transactionlogsareshippedoverthenetworktoreplicateeverythinginanother,identicalsystemintheotherdata-center.Somedistributeddatastoreshavebuilt-insupportformultipledata-centerawarenessandcanreplicatebetweendata-centersinafullyautomaticfashion.

ArangoDB3.3takesanevolutionarystepforwardbyintroducingmulti-data-centersupport,whichisasynchronousdata-centertodata-centerreplication.Oursolutionisasynchronousandscalestoarbitraryclustersizes,providedyournetworklinkbetweenthedata-centershasenoughbandwidth.Itisfault-tolerantwithoutasinglepointoffailureandincludesalotofmetricsformonitoringinaproductionscenario.

DC2DCisavailableintheEnterpriseedition.

Encryptedbackups

ArangodumpcannowcreateencryptedbackupsusingAES256forencryption.Theencryptionkeycanbereadfromafileorfromageneratorprogram.Itworksinsingleserverandclustermode.

Examplefornon-encryptedbackup(everyonewithaccesstothebackupwillbeabletoreadit):

arangodump--collection"secret"dump

Inordertocreateanencryptedbackup,addthe--encryption.keyfileoptionwheninvokingarangodump:

arangodump--collection"secret"dump--encryption.keyfile~/SECRET-KEY

Thekeymustbeexactly32byteslong(requiredbytheAESblockcipher).

Notethatarangodumpwillnotstorethekeyanywhere.Itistheresponsibilityoftheusertofindasafeplaceforthekey.However,arangodumpwillstoretheusedencryptionmethodinafilenamedENCRYPTIONinthedumpdirectory.Thatwayarangorestorecanlaterfindoutwhetheritisdealingwithanencrypteddumpornot.

Tryingtorestoretheencrypteddumpwithoutspecifyingthekeywillfail:

arangorestore--collection"secret-collection"dump--create-collectiontrue

arangorestorewillcomplainwith:

thedumpdataseemstobeencryptedwithaes-256-ctr,butnokeyinformationwasspecifiedtodecryptthedumpitisrecommendedtospecifyeither--encryption.key-fileor--encryption.key-generatorwheninvokingarangorestorewithanencrypteddump

Itisrequiredtousetheexactsamekeywhenrestoringthedata.Againthisisdonebyprovidingthe--encryption.keyfileparameter:

arangorestore--collection"secret-collection"dump--create-collectiontrue--encryption.keyfile~/SECRET-KEY

WhatsNewin3.3

528

Usingadifferentkeywillleadtothebackupbeingnon-recoverable.

NotethatencryptedbackupscanbeusedtogetherwiththealreadyexistingRocksDBencryption-at-restfeature,buttheycanalsobeusedfortheMMFilesengine,whichdoesnothaveencryption-at-rest.

EncryptedbackupsareavailableintheEnterpriseedition.

Server-levelreplicationArangoDBsupportsasynchronousreplicationfunctionalitysinceversion1.4,butreplicatingfromamasterserverwithmultipledatabasesrequiredmanualsetupontheslaveforeachindividualdatabasetoreplicate.Whenanewdatabasewascreatedonthemaster,oneneededtotakeactionontheslavetoensurethatdataforthatdatabasegotactuallyreplicated.Replicationontheslavealsowasnotawareofwhenadatabasewasdroppedonthemaster.

3.3addsserver-levelreplication,whichwillreplicatethecurrentandfuturedatabasesfromthemastertotheslaveautomaticallyaftertheinitialsetup.

Inordertosetupglobalreplicationona3.3slaveforalldatabasesofagiven3.3master,thereisnowtheso-calledglobalApplier.Ithasthesameinterfaceastheexistingapplier,butitwillreplicatefromalldatabasesofthemasterandnotjustasingleone.

Inordertostartthereplicationontheslaveandmakeitreplicatealldatabasesfromagivenmaster,usethesecommandsontheslave:


replication.setupReplicationGlobal({

endpoint:"tcp://127.0.0.1:8529",

username:"root",

password:"",

autoStart:true

});

Tocheckiftheapplierisrunning,alsousetheglobalApplierobject:

replication.globalApplier.state().state

Theserver-levelreplicationrequiresboththemasterandslaveserverstoArangoDBversion3.3orhigher.

Asynchronousfailover

Aresilientsetupcannoweasilybeachievedbyrunningapairofconnectedservers,ofwhichoneinstancebecomesthemasterandtheotheranasynchronouslyreplicatingslave,withautomaticfailoverbetweenthem.

Twoserversareconnectedviaasynchronousreplication.Oneoftheserversiselectedleader,andtheotheroneismadeafollowerautomatically.Atstartup,thetwoserversfightforleadership.Thefollowerwillautomaticallystartreplicationfromthemasterforallavailabledatabases,usingtheserver-levelreplicationintroducedin3.3.

Whenthemastergoesdown,thisisautomaticallydetectedbyanagencyinstance,whichisalsostartedinthismode.Thisinstancewillmakethepreviousfollowerstopitsreplicationandmakeitthenewleader.

Thefollowerwillautomaticallydenyallreadandwriterequestsfromclientapplications.Onlythereplicationitselfisallowedtoaccessthefollower'sdatauntilthefollowerbecomesanewleader.

Whensendingarequesttoreadorwritedataonafollower,thefollowerwillalwaysrespondwithHTTP503(Serviceunavailable)andprovidetheaddressofthecurrentleader.Clientapplicationsanddriverscanusethisinformationtothenmakeafollow-uprequesttotheproperleader:

HTTP/1.1503ServiceUnavailable

X-Arango-Endpoint:http://[::1]:8531

....

WhatsNewin3.3

529

Clientapplicationscanalsodetectwhothecurrentleaderandthefollowersarebycallingthe/_api/cluster/endpointsRESTAPI.ThisAPIisaccessibleonleadersandfollowersalike.

TheArangoDBstartersupportsstartingtwoserverswithasynchronousreplicationandfailoveroutofthebox.

ThearangojsdriverforJavaScript,theGodriverandtheJavadriverforArangoDBsupportautomaticfailoverincasethecurrentlyaccessedserverendpointrespondswithHTTP503.

Blogarticles:

UsingAutomaticfailoverforresilientsingleserversIntroducingthenewArangoDBJavadriverwithloadbalancingandadvancedfallback

RocksDBthrottlingArangoDB3.3allowswriteoperationstotheRocksDBenginebethrottled,inordertopreventlongerwritestalls.Thethrottlingisadaptive,meaningthatitautomaticallyadaptstotheactualwriterate.Thisresultsinmuchmorestableresponsetimes,whichisbetterforclientapplicationsandclusterhealthtests,becausetimeoutscausedbywritestallsarelesslikelytooccurandtheserverthusnotmistakenlyassumedtobedown.

Blogarticle:RocksDBsmoothingforArangoDBcustomers

Fastershardcreationincluster

Whenusingacluster,onenormallywantsresilience,soreplicationFactorissettoatleast2.Thenumberofshardsisoftensettoratherhighvalueswhencreatingcollections.

Creatingacollectionintheclusterwillmakethecoordinatorstorethesetupmetadataofthenewcollectionintheagencyfirst.Subsequentiallyalldatabaseserversoftheclusterwilldetectthatthereisworktodoandwillbegincreatingtheshards.Thiswillfirsthappenfortheshardleaders.Foreachshardleaderthatfinisheswiththesetup,thesynchronousreplicationwithitsfollowersisthenestablished.Thatwillmakesurethateveryfuturedatamodificationwillnotbecomeeffectiveontheleaderonly,butalsoonallthefollowers.

In3.3thissetupprotocolhasgotsomeshortcutsfortheinitialshardcreation,whichspeedsupcollectioncreationbyroughly50to60percent.

LDAPauthenticationTheLDAPauthenticationmoduleintheEnterpriseeditionhasbeenenhanced.Thefollowingoptionshavebeenaddedtoit:

theoption--server.local-authenticationcontrolswhetherthelocal_userscollectionisalsousedforlookingupusers.Thisisalsothedefaultbehavior.IftheauthenticationshallberestrictedtojusttheLDAPdirectory,theoptioncanbesettotrue,andarangodwillthennotmakeanyqueriestoits_userscollectionwhenlookingupusers.

theoption--server.authentication-timeoutcontrolstheexpirationtimeforcachedLDAPuserinformationentriesinarangod.

basicrolesupporthasbeenaddedfortheLDAPmoduleintheEnterpriseedition.NewconfigurationoptionsforLDAPin3.3are:

--ldap.roles-attribute-name

--ldap.roles-transformation

--ldap.roles-search

--ldap.roles-include

--ldap.roles-exclude

--ldap.superuser-role

PleaserefertoLDAPforadetailedexplanation.

Miscellaneousfeatures

WhatsNewin3.3

530

https://www.arangodb.com/2017/10/milestone2-arangodb33-new-data-replication/#using-automatic-failover

https://www.arangodb.com/2017/12/introducing-the-new-arangodb-java-driver-load-balancing/

https://www.arangodb.com/2017/11/rocksdb-smoothing-arangodb-customers/

whencreatingacollectioninthecluster,thereisnowanoptionalparameterenforceReplicationFactor:whenset,thisparameterenforcesthatthecollectionwillonlybecreatediftherearenotenoughdatabaseserversavailableforthedesiredreplicationFactor.

AQLDISTINCTisnotchangingtheorderofprevious(sorted)results

PreviouslytheimplementationofAQLdistinctstoredallencounteredvaluesinahashtableinternally.Whendone,thefinalresultswerereturnedintheorderdictatedbythehashtablethatwasusedtostorethekeys.Thisorderwasmoreorlessunpredictable.Thoughthiswasdocumentedbehavior,itwasinconvenientforendusers.

3.3nowdoesnotchangethesortorderoftheresultanymorewhenDISTINCTisused.

SeveralAQLfunctionshavebeenimplementedinC++,whichcanhelpsavememoryandCPUtimeforconvertingthefunctionargumentsandresults.Thefollowingfunctionshavebeenported:

LEFTRIGHTSUBSTRINGTRIMMATCHES

TheArangoShellpromptsubstitutioncharactershavebeenextended.Nowthefollowingextrasubstitutionscanbeusedforthearangoshprompt:

'%t':currenttimeastimestamp'%a':elpasedtimesinceArangoShellstartinseconds'%p':durationoflastcommandinseconds

Forexample,toshowtheexecutiontimeofthelastcommandexecutedinarangoshintheshell'sprompt,startarangoshusing:

arangosh--console.prompt"%E@%d%p>"

Therearenewstartupoptionsfortheloggingtoaiddebugginganderrorreporting:

--log.role:willshowone-lettercodeofserverrole(A=agent,C=coordinator,...)Thisisespeciallyusefulwhenaggregatinglogs.

Theexistingrolesusedinlogsare:

U:undefined/unclear(usedatstartup)S:singleserverC:coordinatorP:primaryA:agent

--log.line-numbertrue:thisoptionwillnowadditionallyshowthenameoftheC++functionthattriggeredthelogmessage(filenameandlinenumberwerealreadyloggedinpreviousversions)

--log.thread-nametrue:thisnewoptionwilllogthenameoftheArangoDBthreadthattriggeredthelogmessage.WillhavemeaningfuloutputonLinuxonly

maketheArangoShell(arangosh)refillitscollectioncachewhenayet-unknowncollectionisfirstaccessed.Thisfixesthefollowingproblemwhenworkingwiththeshellwhileinanothershellorbyanotherprocessanewcollectionisadded:

arangosh1>db._collections();//shell1listsallcollections

arangosh2>db._create("test");//shell2nowcreatesanewcollection'test'

arangosh1>db.test.insert({});//shell1isnotawareofthecollectioncreated

//inshell2,sotheinsertwillfail

WhatsNewin3.3

531

IncompatiblechangesinArangoDB3.3ItisrecommendedtocheckthefollowinglistofincompatiblechangesbeforeupgradingtoArangoDB3.3,andadjustanyclientprogramsifnecessary.

ThefollowingincompatiblechangeshavebeenmadeinArangoDB3.3:

AQL:duringatraversalifavertexisnotfound,arangodwillnotloganerrorandcontinuewithaNULLvalue,butwillinsteadregisterawarningatthequeryandcontinuewithaNULLvalue.

Ifanon-existingvertexisreferencedfromatraversal,itisnotdesirabletologerrorsasArangoDBcanstoreedgespointingtonon-existingvertices(whichisperfectlyvalidifthelow-levelinsertAPIsareused).Aslinkingtonon-existingverticesmayindicateanissuein/withthedatamodelortheclientapplication,thewarningisregisteredinthequerysoclientapplicationshaveaccesstoit.

ArangoDBusernamesmustnotstartwiththestring:role:.

Thestartupconfigurationparameter--cluster.my-iddoesnothaveanyeffectin3.3.Forcompatibilityreasons,ArangoDB3.3willnotfailonstartupiftheoptionisstillusedintheconfiguration,butitwillsilentlyignorethisoption.

Thestartupconfigurationparameter--cluster.my-local-infoisdeprecatednow.Usingitwillmakearangodlogawarningonstartup.

Serverstartup:therecommendedvaluefortheLinuxkernelsettingin/proc/sys/vm/max_map_countwasincreasedtoavalueeighttimesashighasin3.2.arangodcomparesatstartupiftheeffectivevalueofthissettingispresumablytoolow,anditwillissueawarninginthiscase,recommendingtoincreasethevalue.

Thisisnowmorelikelytohappenthaninpreviousversions,astherecommendedvalueisnoweighttimeshigherthanin3.2.Thestartupwarningswilllooklikethis(withactualnumbersvarying):

WARNING{memory}maximumnumberofmemorymappingsperprocessis65530,whichseemstoolow.itisrecommendedtosetit

toatleast512000

PleaserefertotheLinuxkerneldocumentationformoreinformationonthissetting.ThischangeonlyaffectstheLinuxversionofArangoDB.

Clienttools

Theoption--recycle-idshasbeenremovedfromthearangorestorecommand.Usingthisoptioncouldhaveledtoproblemsontherestore,withpotentialidconflictsbetweentheoriginatingserver(thesourcedumpserver)andthetargetserver(therestoreserver).

Theoption--compatparameterhasbeenremovedfromthearangodumpcommandandthe/_api/replication/dumpRESTAPIendpoint.InordertocreateadumpfromanArangoDB2.8instance,pleaseuseanolderversionoftheclienttools.OlderArangoDBversionsarenolongerbesupportedbythearangodumpandarangorestorebinariesshippedwith3.3.

Miscellaneous

TheminimumsupportedcompilerforcompilingArangoDBfromsourceisnowg++5.4(bumpedupfromg++4.9).ThischangeonlyaffectsusersthatcompileArangoDBontheirown.


532



Storageengines

ArangoDB3.2offerstwostorageengines:

thealways-existingmemory-mappedfilesstorageengineanewstorageenginebasedonRocksDB

Memory-mappedfilesstorageengine(MMFiles)

Theformerstorageengine(namedMMFilesenginehenceforth)persistsdatainmemory-mappedfiles.

Anydatachangesaredonefirstintheengine'swrite-aheadlog(WAL).TheWALisreplayedafteracrashsotheengineoffersdurabilityandcrash-safety.DatafromtheWALiseventuallymovedtocollection-specificdatafiles.Thefilesarealwayswritteninanappend-onlyfashion,sodatainfilesisneveroverwritten.Obsoletedatainfileswilleventuallybepurgedbybackgroundcompactionthreads.

Mostofthisengine'sindexesarebuiltinRAM.Whenacollectionisloaded,thisrequiresrebuildingtheindexesinRAMfromthedatastoredondisk.TheMMFilesenginehascollection-levellocking.

Thisstorageengineisagoodchoicewhendata(includingtheindexes)canfitintheserver'savailableRAM.Ifthesizeofdataplusthein-memoryindexesexceedsthesizeoftheavailableRAM,thenthisenginemaytrytoallocatemorememorythanavailable.Thiswilleithermaketheoperatingsystemswapoutpartsofthedata(andcausediskI/O)or,whennoswapspaceisconfigured,invoketheoperatingsystem'sout-of-memoryprocesskiller.

Thelockingstrategyallowsparallelreadsandisoftengoodenoughinread-mostlyworkloads.Writesneedexclusivelocksonthecollections,sotheycanblockotheroperationsinthesamecollection.Thelockingstrategyalsoprovidestransactionalconsistencyandisolation.

RocksDBstorageengine

TheRocksDBstorageengineisnewinArangoDB3.2.Itisdesignedtostoredatasetsthatarebiggerthantheserver'savailableRAM.Itpersistsalldata(includingtheindexes)inaRocksDBinstance.

ThatmeansanydocumentreadorwriteoperationswillbeansweredbyRocksDBunderthehood.RocksDBwillservethedatafromitsownin-RAMcachesorfromdisk.TheRocksDBenginehasawrite-aheadlog(WAL)andusesbackgroundthreadsforcompaction.Itsupportsdatacompression.

TheRocksDBstorageenginehasdocument-levellocking.Readoperationsdonotblockandareneverblockedbyotheroperations.Writeoperationsonlyblockwritesonthesamedocuments/indexvalues.Becausemultiplewriterscanoperateinparallelonthesamecollection,thereisthepossibilityofwrite-writeconflicts.Ifsuchwriteconflictisdetected,oneofthewriteoperationsisabortedwitherror1200("conflict").Clientapplicationscantheneitheraborttheoperationorretry,basedontherequiredconsistencysemantics.

Storageengineselection

ThestorageenginetouseinanArangoDBclusterorasingle-serverinstancemustbeselectedinitially.ThedefaultstorageengineinArangoDB3.2istheMMFilesengineifnostorageengineisselectedexplicitly.Thisensuresallusersupgradingfromearlierversionscancontinuewiththewell-knownMMFilesengine.

Toselectthestorage-engine,thereistheconfigurationoption--server.storage-engine.Itcanbesettoeithermmfiles,rocksdborauto.Whilethefirsttwovalueswillexplicitlyselectastorageengine,theautooptionwillautomaticallychoosethestorageenginebasedonwhichstorageenginewaspreviouslyselected.Ifnoenginewasselectedpreviously,autowillselecttheMMFilesengine.Ifanenginewaspreviouslyselected,theselectionwillbewrittentoafileENGINEintheserver'sdatabasedirectoryandwillbereadfromthereatanysubsequentserverstarts.

WhatsNewin3.2

533

https://www.github.com/facebook/rocksdb/

Oncethestorageenginewasselected,theselectioncannotbechangedbyadjusting--server.storage-engine.Inordertoswitchtoanotherstorageengine,itisrequiredtore-starttheserverwithanother(empty)databasedirectory.Inordertousedatacreatedwiththeotherstorageengine,itisrequiredtodumpthedatafirstwiththeoldengineandrestoreitusingthenewstorageengine.Thiscanbeachievedviainvokingarangodumpandarangorestore.

UnlikeinMySQL,thestorageengineselectioninArangoDBisforanentireclusteroranentiresingle-serverinstance.Alldatabasesandcollectionswillusethesamestorageengine.

RocksDBstorageengine:supportedindextypes

TheexistingindexesintheRocksDBengineareallpersistent.Thefollowingindexesaresupportedthere:

primary:thistypeofindexisautomaticallycreated.Itindexes_id/_key

edge:thisindexisautomaticallycreatedforedgecollections.Itindexes_fromand_to

hash,skiplist,persistent:theseareuser-definedindexes,Despitetheirnames,theyareneitherhashnorskiplistindexes.TheseindextypesmaptothesameRocksDB-basedsortedindeximplementation.Thesameistrueforthe"persistent"index.Thenames"hash","skiplist"and"persistent"areonlyusedforcompatibilitywiththeMMFilesenginewheretheseindexesexistedinpreviousandthecurrentversionofArangoDB.

geo:user-definedindexforproximitysearches

fulltext:user-definedsortedrevertedindexonwordsoccurringindocuments


WithSatelliteCollections,youcandefinecollectionstoshardtoaclusterandcollectionstoreplicatetoeachmachine.TheArangoDBqueryoptimizerknowswhereeachshardislocatedandsendstherequeststotheDBServersinvolved,whichthenexecutesthequery,locally.Withthisapproach,networkhopsduringjoinoperationsonshardedcollectionscanbeavoidedandresponsetimescanbeclosetothatofasingleinstance.

SatellitecollectionsareavailableintheEnterpriseedition.

MemorymanagementmakearangodstartwithlessV8JavaScriptcontexts

Thisspeedsuptheserverstartandmakesarangoduselessmemoryatstart.WheneveraV8contextisneededbyaFoxxactionorsomeotherJavaScriptoperationandthereisnousableV8context,anewcontextwillbecreateddynamicallynow.

Upto--javascript.v8-contextsV8contextswillbecreated,sothisoptionwillchangeitsmeaning.PreviouslyasmanyV8contextsasspecifiedbythisoptionwerecreatedatserverstart,andthenumberofV8contextsdidnotchangeatruntime.NowuptothisnumberofV8contextswillbeinuseatthesametime,buttheactualnumberofV8contextsisdynamic.

ThegarbagecollectorthreadwillautomaticallydeleteunusedV8contextsafterawhile.Thenumberofsparecontextswillgodowntoasfewasconfiguredinthenewoption--javascript.v8-contexts-minimum.ActuallythatmanyV8contextsarealsocreatedatserverstart.

ThefirstfewrequestsinnewV8contextsmaytakelongerthanincontextsthathavebeentherealready.PerformancemaythereforesufferabitfortheinitialrequestssenttoArangoDBorwhenthereareonlyfewbutperformance-criticalsituationsinwhichnewV8contextsneedtobecreated.Ifthisisaconcern,itcaneasilybefixedbysetting--javascipt.v8-contexts-minimumand--javascript.v8-contextstoarelativelyhighvalue,whichwillguaranteethatmanynumberofV8contextstobecreatedatstartupandkeptaroundevenwhenunused.

WaitingforanunusedV8contextwillnowalsoabortandwritealogmessageincasenoV8contextcanbeacquired/createdafter60seconds.

thenumberofpendingoperationsinarangodcannowbelimitedtoaconfigurablenumber.Ifthisnumberisexceeded,theserverwillnowrespondwithHTTP503(serviceunavailable).Themaximumsizeofpendingoperationsiscontrolledviathestartupoption--server.maximal-queue-size.Settingitto0means"nolimit".

WhatsNewin3.2

534

thein-memorydocumentrevisionscachewasremovedentirelybecauseitdidnotprovidetheexpectedbenefits.The3.1implementationshadoweddocumentdatainRAM,whichincreasedtheserver'sRAMusagebutdidnotspeedupdocumentlookupstoomuch.

Thisalsoobsoletesthestartupoptions--database.revision-cache-chunk-sizeand--database.revision-cache-target-size.

TheMMFilesenginenowdoesnotuseadocumentrevisionscachebuthasin-memoryindexesandmapsdocumentstoRAMautomaticallyviammapwhendocumentsareaccessed.TheRocksDBenginehasitsownmechanismforcachingaccesseddocuments.

CommunicationLayerHTTPresponsesreturnedbyarangodwillnowincludetheextraHTTPheaderx-content-type-options:nosnifftoworkaroundacross-sitescriptingbuginMSIE

thedefaultvaluefor--ssl.protocolwaschangedfromTLSv1toTLSv1.2.Whennotexplicitlyset,arangodandallclienttoolswillnowuseTLSv1.2.

theJSONdatainallincomingHTTPrequestsinnowvalidatedforduplicateattributenames.

IncomingJSONdatawithduplicateattributenameswillnowberejectedasinvalid.PreviousversionsofArangoDBonlyvalidatedtheuniquenessofattributenamesinsideincomingJSONforsomeAPIendpoints,butnotconsistentlyforallAPIs.

InternalJavaScriptRESTactionswillnowhidetheirstacktracestotheclientunlessinHTTPresponses.Insteadtheywillalwayslogtothelogfile.

JavaScript

updatedV8versionto5.7.0.0

changeundocumentedbehaviourincaseofinvalidrevisionidsinIf-MatchandIf-None-Matchheadersfrom400(BAD)to412(PRECONDITIONFAILED).

changedefaultstringtruncationlengthfrom80charactersto256charactersforprint/printShellfunctionsinArangoShellandarangod.Thiswillemitlongerprefixesofstringvaluesbeforetruncatingthemwith...,whichishelpfulfordebugging.ThischangeismostlyusefulwhenusingtheArangoShell(arangosh).

the@arangodbmodulenowprovidesatimefunctionwhichreturnsthecurrenttimeinsecondsasafloatingpointvaluewithmicrosecondprecision.

FoxxThereisnowanofficialHTTPAPIformanagingservices,allowingservicestobeinstalled,modified,uninstalledandreconfiguredwithouttheadministrativewebinterface.

ItisnowpossibletouploadasingleJavaScriptfileinsteadofaziparchiveifyourservicerequiresnoconfiguration,additionalfilesorsetup.Aminimalmanifestwillbegeneratedautomaticallyuponinstallationandtheuploadedfilewillbeusedastheservice'smainentrypoint.

DistributedGraphProcessing

WeaddedsupportforexecutingdistributedgraphalgorithmsakaPregel.Userscanrunarbitraryalgorithmsonanentiregraph,includinginclustermode.Weimplementedanumberofalgorithmsforvariouswell-knowngraphmeasures:

ConnectedComponentsPageRankShortestPathsCentralityMeasures(CentralityandBetweeness)

WhatsNewin3.2

535

CommunityDetection(viaLabelPropagation,Speakers-ListenersLabelPropagationorDMID)Userscancontributetheirownalgorithms

AQL

Optimizerimprovements

GeoindexesarenowimplicitlyandautomaticallyusedwhenusingappropriateSORT/FILTERstatementsinAQL,withouttheneedtousethesomewhatlimitedspecial-purposegeoAQLfunctionsNEARorWITHIN.

ComparedtousingthespecialpurposeAQLfunctionsthisapproachhastheadvantagethatitismorecomposable,andwillalsohonoranyLIMITvaluesusedintheAQLquery.

ThespecialpurposeNEARAQLfunctioncannowbesubstitutedwiththefollowingAQL(providedthereisageoindexpresentonthedoc.latitudeanddoc.longitudeattributes):

FORdocingeoSort

SORTDISTANCE(doc.latitude,doc.longitude,0,0)

LIMIT5

RETURNdoc

WITHINcanbesubstitutedwiththefollowingAQL:

FORdocingeoFilter

FILTERDISTANCE(doc.latitude,doc.longitude,0,0)<2000

RETURNdoc

Miscellaneousimprovements

addedREGEX_REPLACEAQLfunction

REGEX_REPLACE(text,search,replacement,caseInsensitive)→string

Replacethepatternsearchwiththestringreplacementinthestringtext,usingregularexpressionmatching.

text(string):thestringtosearchinsearch(string):aregularexpressionsearchpatternreplacement(string):thestringtoreplacethesearchpatternwithreturnsstring(string):thestringtextwiththesearchregexpatternreplacedwiththereplacementstringwhereverthepatternexistsintext

addednewstartupoption--query.fail-on-warningtomakeAQLqueriesabortinsteadofcontinuingwithwarnings.

Whensettotrue,thiswillmakeanAQLquerythrowanexceptionandabortincaseawarningoccurs.Thisoptionshouldbeusedindevelopmenttocatcherrorsearly.Ifsettofalse,warningswillnotbepropagatedtoexceptionsandwillbereturnedwiththequeryresults.Thestartupoptioncanalsobeoverridenonaperquery-level.

theslowquerylistnowcontainsthevaluesofbindvariablesusedintheslowqueries.Bindvariablesarealsoprovidedforthecurrentlyrunningqueries.Thishelpsdebuggingsloworblockingqueriesthatusedynamiccollectionnamesviabindparameters.

AQLbreakingchangeincluster:TheSHORTEST_PATHstatementusingedgecollectionnamesinsteadofagraphnamesnowrequirestoexplicitlynamethevertexcollectionnameswithintheAQLqueryinthecluster.ItcanbedonebyaddingWITH<name>atthebeginningofthequery.

Example:

FORv,eINOUTBOUNDSHORTEST_PATH@startTO@targetedges[...]

Nowhastobe:

WITHvertices


WhatsNewin3.2

536

Thischangeisduetoavoiddeadlocksitationsinclusteredcase.Anerrorstatingtheaboveisincluded.

Clienttools

addeddataexporttool,arangoexport.

arangoexportcanbeusedtoexportcollectionstojson,jsonlorxmlandexportagraphorcollectionstoxgmml.

added"jsonl"asinputfiletypeforarangoimp

added--translateoptionforarangoimptotranslateattributenamesfromtheinputfilestoattriubtenamesexpectedbyArangoDB

The--translateoptioncanbespecifiedmultipletimes(oncepertranslationtobeexecuted).Thefollowingexamplerenamesthe"id"columnfromtheinputfileto"_key",andthe"from"columnto"_from",andthe"to"columnto"_to":

arangoimp--typecsv--filedata.csv--translate"id=_key"--translate"from=_from"--translate"to=_to"

--translateworksforCSVandTSVinputsonly.

added--threadsoptiontoarangoimptospecifythenumberofparallelimportthreads

changeddefaultvalueforclienttoolsoption--server.max-packet-sizefrom128MBto256MB.thisallowstransferringbiggerresultsetsfromtheserverwithouttheclienttoolsrejectingthemasinvalid.

Authentication

addedLDAPauthentication(Enterpriseonly)

Authorization

addedreadonlymodeforuserscollectionlevelauthorizationrights

Readmoreintheoverview.

Foxxthecookiesessiontransportnowsupportsalloptionssupportedbythecookiemethodoftheresponseobject.

it'snowpossibletoprovideyourownversionofthegraphql-syncmodulewhenusingtheGraphQLextensionsforFoxxbypassingacopyofthemoduleusingthenewgraphqloption.

customAPIendpointscannowbetaggedusingthetagmethodtogenerateacleanerSwaggerdocumentation.

MiscellaneousChanges

arangodnowvalidatesseveralOS/environmentsettingsonstartupandwarnsifthesettingsarenon-ideal.Itadditionallywillprintoutwaystoremedytheoptions.

MostofthechecksareexecutedonLinuxsystemsonly.

added"deduplicate"attributeforarrayindexes,whichcontrolswhetherinsertingduplicateindexvaluesfromthesamedocumentintoauniquearrayindexwillleadtoanerrorornot:

//withdeduplicate=true,whichisthedefaultvalue:

db._create("test");

db.test.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:true});

db.test.insert({tags:["a","b"]});

WhatsNewin3.2

537

db.test.insert({tags:["c","d","c"]});//willwork,becausededuplicate=true

db.test.insert({tags:["a"]});//willfail

//withdeduplicate=false

db._create("test");

db.test.ensureIndex({type:"hash",fields:["tags[*]"],deduplicate:false});

db.test.insert({tags:["a","b"]});

db.test.insert({tags:["c","d","c"]});//willnotwork,becausededuplicate=false

db.test.insert({tags:["a"]});//willfail

WhatsNewin3.2

538

KnownIssuesThefollowingknownissuesarepresentinthisversionofArangoDBandwillbefixedinfollow-upreleases:

RocksDBstorageengine

TheRocksDBstorageengineisintentionallymissingthefollowingfeaturesthatarepresentintheMMFilesengine:

thedatafiledebugger(arango-dfdb)cannotbeusedwiththisstorageengine

RocksDBhasitsowncrashrecoverysousingthedfdbwillnotmakeanysensehere.

APIsthatreturncollectionpropertiesorfigureswillreturnslightlydifferentattributesfortheRocksDBenginethanfortheMMFilesengine.Forexample,theattributesjournalSize,doCompact,indexBucketsandisVolatilearepresentintheMMFilesenginebutnotintheRocksDBengine.ThememoryusagefiguresreportedforcollectionsintheRocksDBengineareestimatevalues,whereastheyareexactfortheMMFilesengine.

theRocksDBenginedoesnotsupportsomeoperationswhichonlymakesenseinthecontextoftheMMFilesengine.Theseare:

therotatemethodoncollectionstheflushmethodforWALfiles

theRocksDBstorageenginedoesnotsupportvolatilecollections

transactionsarelimitedinsize.Transactionsthatgettoobig(intermsofnumberofoperationsinvolvedorthetotalsizeofdatamodifiedbythetransaction)willbecommittedautomatically.EffectivelythismeansthatbigusertransactionsaresplitintomultiplesmallerRocksDBtransactionsthatarecommittedindividually.TheentireusertransactionwillnotnecessarilyhaveACIDpropertiesinthiscase.

Thethresholdvaluesfortransactionsizescanbeconfiguredgloballyusingthestartupoptions

--rocksdb.intermediate-commit-size:ifthesizeofalloperationsinatransactionreachesthisthreshold,thetransactioniscommittedautomaticallyandanewtransactionisstarted.Thevalueisspecifiedinbytes.

--rocksdb.intermediate-commit-count:ifthenumberofoperationsinatransactionreachesthisvalue,thetransactioniscommittedautomaticallyandanewtransactionisstarted.

--rocksdb.max-transaction-size:thisisanupperlimitforthetotalnumberofbytesofalloperationsinatransaction.Iftheoperationsinatransactionconsumemorethanthisthresholdvalue,thetransactionwillautomaticallyabortwitherror32("resourcelimitexceeded").

Itisalsopossibletooverridethesethresholdspertransaction.

Thefollowingknownissueswillberesolvedinfuturereleases:

theRocksDBengineisnotyetperformance-optimizedandpotentiallynotwellconfigured

collectionsforwhichageoindexispresentwillusecollection-levelwritelocksevenwiththeRocksDBengine.Readsfromthesecollectionscanstillbedoneinparallelbutnowrites

modifyingdocumentsinacollectionwithageoindexwillcausemultipleadditionalwritestoRocksDBformaintainingtheindexstructures

thenumberofdocumentsreportedforcollections(db.<collection>.count())maybeslightlywrongduringtransactionsifthereareparalleltransactionsongoingforthesamecollectionthatalsomodifythenumberofdocuments

theanyoperationtoprovidearandomdocumentfromacollectionissupportedbytheRocksDBenginebuttheoperationhasmuchhigheralgorithmiccomplexitythanintheMMFilesengine.Itisthereforediscouragedtocallitforcasesotherthanmanualinspectionofafewdocumentsinacollection

AQLqueriesintheclusterstillissueanextralockingHTTPrequestpershardthoughthiswouldnotbenecessaryfortheRocksDBengineinmostcases

Installer

KnownIssuesin3.2

539

Upgradingfrom3.1to3.2onWindowsrequirestheusertomanuallycopythedatabasedirectorytothenewlocationandrunanupgradeonthedatabase.PleaseconsulttheDocumentationfordetailedinstructions.

SystemIntegration

OnsomeLinuxsystemssystemdandsystemvmightreportthatthearangodbserviceisingoodconditionwhenitcouldnotbestarted.Inthiscasetheuserneedstocheck/var/log/arangodb3forfurtherinformationaboutthefailedstartup.

MacOSX

Storageengineisnotchangeableonanexistingdatabase.Currentlyonlytheinitialselectionofthestorageengineissupported.Inordertouseanotherstorageengine,youhavetodeleteyourArangoDBapplication(MacApplicationFolder)and/Users/<your_user_name>/Library/ArangoDBfolder.

OpenSSL1.1

ArangoDBhasbeentestedwithOpenSSL1.0onlyandwon'tbuildagainst1.1whencompilingonyourown.SeehereforhowtocompileonsystemsthatshipOpenSSL1.1bydefault.

KnownIssuesin3.2

540


AQL

AQLbreakingchangeincluster:TheSHORTEST_PATHstatementusingedge-collectionnamesinsteadofagraphnamenowrequirestoexplicitlynamethevertex-collectionnameswithintheAQLqueryinthecluster.ItcanbedonebyaddingWITH<name>atthebeginningofthequery.

Example:


Nowhastobe:

WITHvertices


Thischangeisduetoavoiddead-locksitationsinclusteredcase.Anerrorstatingtheaboveisincluded.

RESTAPIRemovedundocumentedinternalHTTPAPI:

PUT/_api/edgesThedocumentedGET/_api/edgesandtheundocumentedPOST/_api/edgesremainsunmodified.

changeundocumentedbehaviourincaseofinvalidrevisionidsinIf-MatchandIf-None-MatchheadersfromreturningHTTPstatuscode400(badrequest)toreturningHTTPstatuscode412(preconditionfailed).

theRESTAPIforfetchingthelistofcurrentlyrunningAQLqueriesandtheRESTAPIforfetchingthelistofslowAQLqueriesnowreturnanextrabindVarsattributewhichcontainsthebindparametersusedbythequeries.

ThisaffectsthereturnvaluesofthefollowingAPIendpoints:

GET/_api/query/currentGET/_api/query/slow

TheRESTAPIforretrievingindexes(GET/_api/index)nowreturnsthededuplicateattributeforeachindex

TheRESTAPIforcreatingindexes(POST/_api/index)nowacceptstheoptionaldeduplicateattribute

JavaScriptAPI

changeundocumentedbehaviourincaseofinvalidrevisionidsinJavaScriptdocumentoperationsfromreturningerrorcode1239("illegaldocumentrevision")toreturningerrorcode1200("conflict").

thecollection.getIndexes()functionnowreturnsthededuplicateattributeforeachindex

thecollection.ensureIndex()functionnowacceptstheoptionaldeduplicateattribute

Foxx

JWTtokensissuedbythebuilt-inJWTsessionstoragenowcorrectlyspecifytheiatandexpvaluesinsecondsratherthanmillisecondsasspecifiedintheJSONWebTokenstandard.


541

Thismayresultinpreviouslyexpiredtokensusingmillisecondsbeingincorrectlyaccepted.ForthisreasonitisrecommendedtoreplacethesigningsecretorsetthenewmaxExpoptiontoareasonablevaluethatissmallerthantheoldestissuedexpirationtimestamp.

ForexamplesettingmaxExpto10**12wouldinvalidateallincorrectlyissuedtokensbefore9September2001withoutimpairingnewtokensuntiltheyear33658(atwhichpointthesetokensarehopefullynolongerrelevant).

ArangoDBrunninginstandalonemodewillcommitallservicesinthejavascript.app-pathtothedatabaseonstartup.ThismayresultinuninstalledservicesshowingupinArangoDBiftheywerenotproperlyremovedfromthefilesystem.

ArangoDBcoordinatorsinaclusternowperformaself-healingstepduringstartuptoensureinstalledservicesareconsistentaccrossallcoordinators.WerecommendbackingupyourservicesandconfigurationbeforeupgradingtoArangoDB3.2,especiallyifyouhavemadeuseofthedevelopmentmode.

Servicesinstalledbeforeupgradingto3.2(includingservicesinstalledonalphareleasesofArangoDB3.2)areNOTpickedupbythecoordinatorself-healingwatchdog.Thiscanbesolvedbyeitherupgrading/replacingtheseservicesorbyusingthe"commit"routeoftheFoxxservicemanagementHTTPAPI,whichcommitstheexactservicesinstalledonagivencoordinatortothecluster.Newserviceswillbepickedupautomatically.

TheformatusedbyFoxxtostoreinternalservicemetadatainthedatabasehasbeensimplifiedandexistingdocumentswillbeupdatedtothenewformat.Ifyouhavemadeanychangestothedatastoredinthe_appssystemcollection,youmaywishtoexportthesechangesastheywillbeoverwritten.

ThereisnowanofficialHTTPAPIformanagingservices.IfyouwerepreviouslyusinganyoftheundocumentedAPIsortheroutesusedbytheadministrativewebinterfacewehighlyrecommendmigratingtothenewAPI.TheoldundocumentedHTTPAPIformananagingservicesisdeprecatedandwillberemovedinafutureversionofArangoDB.

Althoughchangestothefilesystemoutsideofdevelopmentmodewerealreadystronglydiscouraged,thisisareminderthattheyarenolongersupported.Allfilesgeneratedbyservices(whetherbyasetupscriptorduringnormaloperationsuchasuploads)shouldeitherbestoredoutsidetheservicedirectoryorbeconsideredextremelyvolatile.

IntroduceddistinctionbetweenarangoUserandauthorizedinFoxxrequests.ClusterinternalrequestswillneverhaveanarangoUserbutareauthorized.InearlierversionsofArangoDBpartsofthestatisticswerenotaccessiblebythecoordinatorsbecausetheunderlyingFoxxservicecouldn'tauthorizetherequests.Itnowcorrectlychecksthenewreq.authorizedproperty.req.arangoUserstillworksasbefore.Endusersmayusethisnewpropertyaswelltoeasilycheckifarequestisauthorizedornotregardlessofaspecificuser.

Command-lineoptionschanged

--server.maximal-queue-sizeisnowanabsolutemaximum.Ifthequeueisfull,then503isreturned.Settingitto0means"nolimit".Thedefaultvalueforthisoptionisnow0.

thedefaultvaluefor--ssl.protocolhasbeenchangedfrom4(TLSv1)to5(TLSv1.2).

thestartupoptions--database.revision-cache-chunk-sizeand--database.revision-cache-target-sizearenowobsoleteanddonothing

thestartupoption--database.index-threadsoptionisnowobsolete

theoption--javascript.v8-contextsisnowanabsolutemaximum.TheservermaystartlessV8contextsforJavaScriptexecutionatstartup.IfatsomepointtheserverneedsmoreV8contextsitmaystartthemdynamically,untilthenumberofV8contextsreachesthevalueof--javascript.v8-contexts.

theminimumnumberofV8contextstocreateatstartupcanbeconfiguredviathenewstartupoption--javascript.v8-contexts-minimum.

addedcommand-lineoption--javascript.allow-admin-execute

Thisoptioncanbeusedtocontrolwhetheruser-definedJavaScriptcodeisallowedtobeexecutedonserverbysendingviaHTTPtotheAPIendpoint/_admin/executewithanauthenticateduseraccount.Thedefaultvalueisfalse,whichdisablestheexecutionofuser-definedcode.Thisisalsotherecommendedsettingforproduction.Intestenvironments,itmaybeconvenienttoturntheoptiononinordertosendarbitrarysetuporteardowncommandsforexecutionontheserver.


542

TheintroductionofthisoptionchangesthedefaultbehaviorofArangoDB3.2:3.2nowbydefaultdisablestheexecutionofJavaScriptcodeviathisAPI,whereasearlierversionsallowedit.Torestoretheoldbehavior,itisnecessarytosettheoptiontotrue.

UsersManagement

Itisnolongersupportedtoaccessthe_userscollecctioninanywaydirectly,exceptthroughtheofficial@arangodb/usersmoduleorthe_apit/usersRESTAPI.

Theaccesstothe_userscollectionfromoutsideofthearangodserverprocessisnowforbidden(Throughdrivers,arangoshortheRESTAPI).Foxxservicesarestillbeabletoaccessthe_userscollectionfornow,butthismightchangeinfutureminorreleases.

Theinternalformatofthedocumentsinthe_userscollectionhaschangedfrompreviousversions

The_queuescollectiononlyallowsread-onlyaccessfromoutsideofthearangodserverprocess.

Accessing_queuesisonlysupportedthroughtheofficial@arangodb/queuesmoduleforFoxxapps.


543


SmartGraphs

ArangoDB3.1addsafirstmajorenterpriseonlyfeaturecalledSmartGraphs.SmartGraphsformanadditiontothealreadyexistinggraphfeaturesandallowtoscalegraphsbeyondasinglemachinewhilekeepingalmostthesamequeryperformance.TheSmartGraphfeatureissuggestedforallgraphdatabaseusecasesthatrequireaclusterofdatabaseserversforwhateverreason.Youcaneitherhaveagraphthatistoolargetobestoredonasinglemachineonly.Oryoucanhaveasmallgraph,butatthesametimeneedadditionaldatawithhastobeshardedandyouwanttokeepalloftheminthesameenvirenment.Oryousimplyusetheclusterforhigh-availability.InalltheabovecasesSmartGraphswillsignificantlyincreasetheperformanceofgraphoperations.Formoredetailedinformationreadthismanualsection.

Dataformat

Theformatoftherevisionvaluesstoredinthe_revattributeofdocumentshasbeenchangedin3.1.Upto3.0theywerestringscontaininglargishdecimalnumbers.With3.1,revisionvaluesarestillstrings,butareactuallyencodedtimestampsofthecreationdateoftherevisionofthedocument.Thetimestampsareacquiredusingahybridlogicalclock(HLC)ontheDBserverthatholdstherevision(fortheconceptofahybridlogicalclockseethispaper).Seethismanualsectionfordetails.

ArangoDB>=3.1canArangoDB3.0databasedirectoriesandwillsimplycontinuetousetheold_revattributevalues.Newrevisionswillbewrittenwiththenewtimestamps.

ItishighlyrecommendedtobackupallyourdatabeforeloadingadatabasedirectorythatwaswrittenbyArangoDB<=3.0intoanArangoDB>=3.1.

CommunicationLayerArangoDBupto3.0usedlibevforthecommunicationlayer.ArangoDBstartingfrom3.1usesBoostASIO.

StartingwithArangoDB3.1webegintoprovidetheVelocyStreamProtocol(vst)asaadditiontotheestablishedhttpprotocol.

Afewoptionshavechangedconcerningcommunication,pleasecheckoutIncompatiblechangesin3.1.

ClusterForitsinternalclustercommunicationa(bundledversion)ofcurlisnowbeingused.Thisenablesasynchronousoperationthroughouttheclusterandshouldimprovegeneralperformanceslightly.

Authenticationisnowsupportedwithinthecluster.

Documentrevisionscache

TheArangoDBservernowprovidesanin-memorycacheforfrequentlyaccesseddocumentrevisions.Documentsthatareaccessedduringread/writeoperationsareloadedintotherevisionscacheautomatically,andsubsequentlyservedfromthere.

Thecachehasatotaltargetsize,whichcanbecontrolledwiththestartupoption--database.revision-cache-target-size.Oncethecachereachesthetargetsize,olderentriesmaybeevictedfromthecachetofreememory.Notethatthetargetsizecurrentlyisahighwatermarkthatwilltriggercachememorygarbagecollectionifexceeded.However,ifallcachechunksarestillinusewhenthehighwatermarkisreached,thecachemaystillgrowandallocatemorechunksuntilcacheentriesbecomeunusedandareallowedtobegarbage-collected.

WhatsNewin3.1

544


http://software.schmorp.de/pkg/libev.html

Thecacheismaintainedonaper-collectionbasis,thatis,memoryforthecacheisallocatedonaper-collectionbasisinchunks.Thesizeforthecachememorychunkscanbecontrolledviathestartupoption--database.revision-cache-chunk-size.Thedefaultvalueis4MBperchunk.Biggerchunksizesallowsavingmoredocumentsperchunk,whichcanleadtomoreefficientchunkallocationandlookups,butwillalsoleadtomemorywasteifmanychunksareallocatedandnotfullyused.Thelatterwillbethecaseifthereexistmanysmallcollectionswhichallallocatetheirownchunksbutnotfullyutilizethembecauseofthelownumberofdocuments.

AQL

Functionsadded

ThefollowingAQLfunctionshavebeenaddedin3.1:

OUTERSECTION(array1,array2,...,arrayn):returnsthevaluesthatoccuronlyonceacrossallarraysspecified.

DISTANCE(lat1,lon1,lat2,lon2):returnsthedistancebetweenthetwocoordinatesspecifiedby(lat1,lon1)and(lat2,lon2).Thedistanceiscalculatedusingthehaversineformula.

JSON_STRINGIFY(value):returnsaJSONstringrepresentationofthevalue.

JSON_PARSE(value):convertsaJSON-encodedstringintoaregularobject

Indexusageintraversals

3.1allowsAQLtraversalstouseotherindexesthanjusttheedgeindex.Traversalswithfiltersonedgescannowmakeuseofmorespecificindexes.Forexample,thequery

FORv,e,pIN2OUTBOUND@start@@edge

FILTERp.edges[0].foo=="bar"

RETURN[v,e,p]

mayuseahashindexon["_from","foo"]insteadoftheedgeindexonjust["_from"].


MaketheAQLqueryoptimizerinjectfilterconditionexpressionsreferredtobyvariablesduringfilterconditionaggregation.Forexample,inthefollowingquery

FORdocINcollection

LETcond1=(doc.value==1)

LETcond2=(doc.value==2)

FILTERcond1||cond2

RETURN{doc,cond1,cond2}

theoptimizerwillnowinjecttheconditionsforcond1andcond2intothefilterconditioncond1||cond2,expandingitto(doc.value==1)||(doc.value==2)andmakingtheseconditionsavailableforindexsearching.

Notethattheoptimizerpreviouslyalreadyinjectedsomeconditionsintootherconditions,butonlyifthevariablethatdefinedtheconditionwasnotusedelsewhere.Forexample,thefilterconditioninthequery

FORdocINcollection

LETcond=(doc.value==1)

FILTERcond

RETURN{doc}

alreadygotoptimizedbeforebecausecondwasonlyusedonceinthequeryandtheoptimizerdecidedtoinjectitintotheplacewhereitwasused.

Thisonlyworkedforvariablesthatwerereferredtoonceinthequery.Whenavariablewasusedmultipletimes,theconditionwasnotinjectedasinthefollowingquery

FORdocINcollection

WhatsNewin3.1

545

LETcond=(doc.value==1)

FILTERcond

RETURN{doc,cond}

3.1allowsusingthisconditionsothatthequerycanuseanindexondoc.value(ifsuchindexexists).


Theperformanceofthe[*]operatorwasimprovedforcasesinwhichthisoperatordidnotuseanyfilters,projectionsand/oroffset/limits.

TheAQLqueryexecutorcannowreportthetimerequiredforloadingandlockingthecollectionsusedinanAQLquery.Whenprofilingisenabled,itwillreportthetotalloadingandlockingtimeforthequeryintheloadingcollectionssub-attributeoftheextra.profilevalueoftheresult.TheloadingandlockingtimecanalsobeviewintheAQLqueryeditorinthewebinterface.

AuditLogAuditlogginghasbeenadded,seeAuditing.

ClienttoolsAddedoption--skip-linesforarangoimpThisallowsskippingthefirstfewlinesfromtheimportfileincasetheCSVorTSVimportareusedandsomeinitiallinesshouldbeskippedfromtheinput.

WebAdminInterfaceTheusabilityoftheAQLeditorsignificantlyimproved.InadditiontothestandardJSONoutput,theAQLEditorisnowabletorenderqueryresultsasagraphprevieworatable.FurthermoretheAQLeditordisplaysqueryprofilinginformation.

AddedanewGraphViewerinordertoexchangethetechnicallyobsoleteversion.ThenewGraphViewerisbasedonCanvasbutdoesalsoincludeafirstWebGLimplementation(limitedfunctionality-willchangeinthefuture).ThenewGraphVieweroffersasmoothwaytodiscoverandvisualizeyourgraphs.

Theshardviewinclustermodenowdisplaysaprogressindicatorwhilemovingshards.

AuthenticationUptoArangoDB3.0authenticationofclientrequestswasonlypossiblewithHTTPbasicauthentication.

Startingwith3.1itisnowpossibletoalsouseaJSONWebTokens(JWT)forauthenticatingincomingrequests.

FordetailschecktheHTTPauthenticationchapter.Bothauthenticationmethodsarevalidandwillbesupportedinthenearfuture.Usewhateversuitsyoubest.

Foxx

GraphQL

ItisnoweasytogetstartedwithprovidingGraphQLAPIsinFoxx,seeFoxxGraphQL.

OAuth2

FoxxnowofficiallyprovidesamoduleforimplementingOAuth2clients,seeFoxxOAuth2.

Per-routemiddleware

WhatsNewin3.1

546

https://jwt.io/

It'snowpossibletospecifymiddlewarefunctionsforaroutewhendefiningaroutehandler.Thesemiddlewarefunctionsonlyapplytothesinglerouteandsharetheroute'sparameterdefinitions.CheckouttheFoxxRouterdocumentationformoreinformation.

WhatsNewin3.1

547


CommunicationLayer

TheinternalcommicationlayerisnowbasedonBoostASIO.Afewoptionsregardingthreadsandcommunicationhavebeenchanged.

Therearenolongertwodifferentthreadspools(--scheduler.threadsand--server.threads).Theoption--scheduler.threadshasbeenremoved.Thenumberofthreadsisnowcontrolledbytheoption--server.threadsonly.Bydefault--server.threadsissettothenumberofhyper-cores.

Asaconsequenceofthechange,thefollowing(hidden)startupoptionshavebeenremoved:

--server.extra-threads

--server.aql-threads

--server.backend

--server.show-backends

--server.thread-affinity

AQL

ThebehavioroftheAQLarraycomparisonoperatorshaschangedforemptyarrays:

ALLandANYnowalwaysreturnfalsewhentheleft-handoperandisanemptyarray.Thebehaviorfornon-emptyarraysdoesnotchange:

[]ALL==1willreturnfalse[1]ALL==1willreturntrue[1,2]ALL==1willreturnfalse[2,2]ALL==1willreturnfalse[]ANY==1willreturnfalse[1]ANY==1willreturntrue[1,2]ANY==1willreturntrue[2,2]ANY==1willreturnfalse

NONEnowalwaysreturnstruewhentheleft-handoperandisanemptyarray.Thebehaviorfornon-emptyarraysdoesnotchange:

[]NONE==1willreturntrue[1]NONE==1willreturnfalse[1,2]NONE==1willreturnfalse[2,2]NONE==1willreturntrue

WITHinclustertraversalsisnowmandatoryinordertoavoiddeadlocks.

Dataformatchanges

TheattributemaximalSizehasbeenrenamedtojournalSizeincollectionmeta-datafiles("parameter.json").FilescontainingthemaximalSizeattributewillstillbepickedupcorrectlyfornot-yetadjustedcollections.

Theformatoftherevisionvaluesstoredinthe_revattributeofdocumentshasbeenchangedin3.1.Upto3.0theywerestringscontaininglargishdecimalnumbers.With3.1,revisionvaluesarestillstrings,butareactuallyencodedtimestampsofthecreationdateoftherevisionofthedocument.Thetimestampsareacquiredusingahybridlogicalclock(HLC)ontheDBserverthatholdstherevision(fortheconceptofahybridlogicalclockseethispaper).Seethismanualsectionfordetails.


548


ArangoDB>=3.1canArangoDB3.0databasedirectoriesandwillsimplycontinuetousetheold_revattributevalues.Newrevisionswillbewrittenwiththenewtimestamps.

ItishighlyrecommendedtobackupallyourdatabeforeloadingadatabasedirectorythatwaswrittenbyArangoDB<=3.0intoanArangoDB>=3.1.

Tochangeallyourold_revattributesintonewstyletimestampsyouhavetousearangodumptodumpalldataout(usingArangoDB3.0),andusearangorestoreintothenewArangoDB3.1,whichisthesafestwaytoupgrade.

Thechangealsoaffectsthereturnformatof_revvaluesandotherrevisionvaluesinHTTPAPIs(seebelow).

HTTPAPIchanges

APIsadded

ThefollowingHTTPRESTAPIshavebeenaddedforonlinelogleveladjustmentoftheserver:

GET/_admin/log/levelreturnsthecurrentloglevelsettingsPUT/_admin/log/levelmodifiesthecurrentloglevelsettings

APIschanged

thefollowingRESTAPIsthatreturnrevisionidsnowmakeuseofthenewrevisionidformatintroducedin3.1.Allrevisionidsreturnedwillbestringsasin3.0,buthaveadifferentinternalformat.

ThefollowingAPIsareaffected:

GET/_api/collection/{collection}/checksum:revisionattributeGET/_api/collection/{collection}/revision:revisionattributeallotherAPIsthatreturndocuments,whichmayincludethedocuments'_revattribute

Clientapplicationsshouldnottrytointerprettheinternalsofrevisionvalues,butonlyuserevisionvaluesforcheckingwhethertworevisionstringsareidentical.

thereplicationRESTAPIswillnowusetheattributenamejournalSizeinsteadofmaximalSizewhenreturninginformationaboutcollections.

thedefaultvalueforkeepNullhasbeenchangedfromfalsetotrueforthefollowingpartialupdateoperationsforverticesandedgesin/_api/gharial:

PATCH/_api/gharial/{graph}/vertex/{collection}/{key}PATCH/_api/gharial/{graph}/edge/{collection}/{key}

ThevalueforkeepNullcanstillbesetexplicitlytofalsebysettingtheURLparameterkeepNulltoavalueoffalse.

theRESTAPIfordroppingcollections(DELETE/_api/collection)nowacceptsanoptionalquerystringparameterisSystem,whichcansettotrueinordertodropsystemcollections.Iftheparameterisnotsetornotsettotrue,theRESTAPIwillrefusetodropsystemcollections.InpreviousversionsofArangoDB,theisSystemparameterdidnotexist,andtherewasnodistinctionbetweensystemandnon-systemcollectionswhendroppingcollections.

theRESTAPIforretrievingAQLqueryresults(POST/_api/cursor)willnowreturnanadditionalsub-attributeloadingcollectionsthatwillcontainthetotaltimerequiredforloadingandlockingcollectionsduringtheAQLquerywhenprofilingisenabled.Theattributecanbefoundintheextraresultattributeinsub-attributeloadingcollections.Theattributewillonlybesetifprofilingwasenabledforthequery.

FoxxTesting

TheQUnitinterfacetoMochahasbeenremoved.Thisaffectsthebehaviourofthesuite,test,before,after,beforeEachandafterEachfunctionsinFoxxtestsuites.ThesuiteandtestfunctionsarenowprovidedbytheTDDinterface.Thebefore,after,beforeEachandafterEachfunctionsarenowprovidedbytheBDDinterface.

Thisshouldnotcauseanyproblemswithexistingtestsbutmayresultinfailuresintestcasesthatpreviouslypassedforthewrongreasons.Specificallytheexecutionorderofthebefore,after,etcfunctionsnowfollowstheintendedorderandisnolongerarbitrary.


549

FordetailsontheexpectedbehaviourofthesefunctionsseethetestingchapterintheFoxxdocumentation.


550


Internaldataformatchanges

ArangoDBnowusesVelocyPackforstoringdocuments,queryresultsandtemporarilycomputedvalues.Usingasingledataformatremovedtheneedforsomedataconversionsinthecorethatslowedoperationsdownpreviously.

TheVelocyPackformatisalsoquitecompact,andreducesstoragespacerequirementsfor"small"valuessuchasboolean,integers,shortstrings.ThiscanspeedupseveraloperationsinsideAQLqueries.

VelocyPackdocumententriesstoredondiskarealsoself-contained,inthesensethateachstoreddocumentwillcontainallofitsdatatypeandattributenamedescriptions.Whilethismayrequireabitmorespaceforstoringthedocuments,itremovestheoverheadoffetchingattributenamesanddocumentlayoutfromsharedstructuresasinpreviousversionsofArangoDB.Italsosimplifiesthecodepathsforstoringandreadingdocuments.

AQLimprovements

Syntaximprovements

LIKEstring-comparisonoperator

AQLnowprovidesaLIKEoperatorandcanbeusedtocomparestringslikethis,forexampleinsidefilterconditions:

valueLIKEsearch

ThischangemakesLIKEanAQLkeyword.UsingLIKEasanattributeorcollectionnameinAQLthusrequiresquotingthenamefromnowon.

TheLIKEoperatoriscurrentlyimplementedbycallingthealreadyexistingAQLfunctionLIKE,whichalsoremainsoperationalin3.0.UsetheLIKEfunctionincaseyouwanttosearchcase-insensitive(optionalparameter),astheLIKEoperatoralwayscomparescase-sensitive.

AQLarraycomparisonoperators

AllAQLcomparisonoperatorsnowalsoexistinanarrayvariant.Inthearrayvariant,theoperatorisprecededwithoneofthekeywordsALL,ANYorNONE.Usingoneofthesekeywordschangestheoperatorbehaviortoexecutethecomparisonoperationforall,any,ornoneofitslefthandargumentvalues.Itisthereforeexpectedthatthelefthandargumentofanarrayoperatorisanarray.

Examples:

[1,2,3]ALLIN[2,3,4]//false

[1,2,3]ALLIN[1,2,3]//true

[1,2,3]NONEIN[3]//false

[1,2,3]NONEIN[23,42]//true

[1,2,3]ANYIN[4,5,6]//false

[1,2,3]ANYIN[1,42]//true

[1,2,3]ANY==2//true

[1,2,3]ANY==4//false

[1,2,3]ANY>0//true

[1,2,3]ANY<=1//true

[1,2,3]NONE<99//false

[1,2,3]NONE>10//true

[1,2,3]ALL>2//false

[1,2,3]ALL>0//true

[1,2,3]ALL>=3//false

["foo","bar"]ALL!="moo"//true

WhatsNewin3.0

551

https://github.com/arangodb/velocypack

["foo","bar"]NONE=="bar"//false

["foo","bar"]ANY=="foo"//true

Regularexpressionstring-comparisonoperators

AQLnowsupportstheoperators=~and!~fortestingstringsagainstregularexpressions.=~testsifastringvaluematchesaregularexpression,and!~testsifastringvaluedoesnotmatcharegularexpression.

Thetwooperatorsexpecttheirleft-handoperandstobestrings,andtheirright-handoperandstobestringscontainingvalidregularexpressionsasspecifiedbelow.

Theregularexpressionsmayconsistofliteralcharactersandthefollowingcharactersandsequences:

.–thedotmatchesanysinglecharacterexceptlineterminators.Toincludelineterminators,use[\s\S]insteadtosimulate.withDOTALLflag.\d–matchesasingledigit,equivalentto[0-9]\s–matchesasinglewhitespacecharacter\S–matchesasinglenon-whitespacecharacter\t–matchesatabcharacter\r–matchesacarriagereturn\n–matchesaline-feedcharacter[xyz]–setofcharacters.matchesanyoftheenclosedcharacters(i.e.x,yorzinthiscase[^xyz]–negatedsetofcharacters.matchesanyothercharacterthantheenclosedones(i.e.anythingbutx,yorzinthiscase)[x-z]–rangeofcharacters.Matchesanyofthecharactersinthespecifiedrange,e.g.[0-9A-F]tomatchanycharacterin0123456789ABCDEF[^x-z]–negatedrangeofcharacters.Matchesanyothercharacterthantheonesspecifiedintherange(xyz)–definesandmatchesapatterngroup(x|y)–matcheseitherxory –matchesthebeginningofthestring(e.g. xyz)$–matchestheendofthestring(e.g.xyz$)

Notethatthecharacters.,*,?,[,],(,),{,}, ,and$haveaspecialmeaninginregularexpressionsandmayneedtobeescapedusingabackslash(\\).Aliteralbackslashshouldalsobeescapedusinganotherbackslash,i.e.\\\\.

Charactersandsequencesmayoptionallyberepeatedusingthefollowingquantifiers:

x*–matcheszeroormoreoccurrencesofxx+–matchesoneormoreoccurrencesofxx?–matchesoneorzerooccurrencesofxx{y}–matchesexactlyyoccurrencesofxx{y,z}–matchesbetweenyandzoccurrencesofxx{y,}–matchesatleastyoccurencesofx

Enclosingidentifiersinforwardticks

AQLidentifierscannowoptionallybeenclosedinforwardticksinadditiontousingbackwardticks.ThisallowsconvenientwritingofAQLqueriesinJavaScripttemplatestrings(whicharedelimitedwithbackticksthemselves),e.g.

varq=`FORdocIN´collection´RETURNdoc.´name´`;

Functionsadded


REGEX_TEST(value,regex):testswhetherthestringvaluematchestheregularexpressionspecifiedinregex.Returnstrueifitmatches,andfalseotherwise.

Thesyntaxforregularexpressionsisthesameasfortheregularexpressionoperators=~and!~.

WhatsNewin3.0

552

HASH(value):Calculatesahashvalueforvalue.valueisnotrequiredtobeastring,butcanhaveanydatatype.Thecalculatedhashvaluewilltakethedatatypeofvalueintoaccount,soforexamplethenumber1andthestring"1" willhavedifferenthashvalues.Forarraysthehashvalueswillbecrearedifthearrayscontainexactlythesamevalues(includingvaluetypes)inthesameorder.Forobjectsthesamehashvalueswillbecreatediftheobjectshaveexactlythesameattributenamesandvalues(includingvaluetypes).Theorderinwhichattributesappearinsideobjectsisnotimportantforhashing.Thehashvaluereturnedbythisfunctionisanumber.ThehashalgorithmisnotguaranteedtoremainthesameinfutureversionsofArangoDB.Thehashvaluesshouldthereforebeusedonlyfortemporarycalculations,e.g.tocompareiftwodocumentsarethesame,orforgroupingvaluesinqueries.

TYPENAME(value):Returnsthedatatypenameofvalue.Thedatatypenamecanbeeithernull,bool,number,string,arrayorobject.

LOG(value):Returnsthenaturallogarithmofvalue.ThebaseisEuler'sconstant(2.71828...).

LOG2(value):Returnsthebase2logarithmofvalue.

LOG10(value):Returnsthebase10logarithmofvalue.

EXP(value):ReturnsEuler'sconstant(2.71828...)raisedtothepowerofvalue.

EXP2(value):Returns2raisedtothepowerofvalue.

SIN(value):Returnsthesineofvalue.

COS(value):Returnsthecosineofvalue.

TAN(value):Returnsthetangentofvalue.

ASIN(value):Returnsthearcsineofvalue.

ACOS(value):Returnsthearccosineofvalue.

ATAN(value):Returnsthearctangentofvalue.

ATAN2(y,x):Returnsthearctangentofthequotientofyandx.

RADIANS(value):Returnstheangleconvertedfromdegreestoradians.

DEGREES(value):Returnstheangleconvertedfromradianstodegrees.


"inline-subqueries"rule

TheAQLoptimizerrule"inline-subqueries"hasbeenadded.ThisrulecanpulloutcertainsubqueriesthatareusedasanoperandtoaFORlooponelevelhigher,eliminatingthesubquerycompletely.Thisreducescomplexityofthequery'sexecutionplanandwilllikelyenablefurtheroptimizations.Forexample,thequery

FORiIN(

FORjIN[1,2,3]

RETURNj

)

RETURNi

willbetransformedbytheruleto:

FORiIN[1,2,3]

RETURNi

Thequery

FORnameIN(

FORdocIN_users

FILTERdoc.status==1

RETURNdoc.name

)

LIMIT2

WhatsNewin3.0

553

RETURNname

willbetransformedinto

FORtmpIN_users

FILTERtmp.status==1

LIMIT2

RETURNtmp.name

TherulewillonlyfirewhenthesubqueryisusedasanoperandtoaFORloop,andifthesubquerydoesnotcontainaCOLLECTwithanINTOvariable.

"remove-unnecessary-calculations"rule

TheAQLoptimizerrule"remove-unnecessary-calculations"nowfiresinmorecasesthaninpreviousversions.Thisruleremovescalculationsfromexecutionplans,andbyhavinglesscalculationsdone,aquerymayexecutefasterorrequireslessmemory.

Therulewillnowremovecalculationsthatareusedexactlyonceinotherexpressions(e.g.LETa=docRETURNa.value)andcalculations,orcalculationsthatarejustreferencestoothervariables(e.g.LETa=b).

"optimize-traversals"rule

TheAQLoptimizerrule"merge-traversal-filter"wasrenamedto"optimize-traversals".TherulewillremoveunusededgeandpathresultvariablesfromthetraversalincasetheyarespecifiedintheFORsectionofthetraversal,butnotreferencedlaterinthequery.Thissavesconstructingedgesandpathsresultsthatarenotusedlater.

AQLnowusesVelocyPackinternallyforstoringintermediatevalues.Formanyvaluetypesitcannowgetawaywithoutextramemoryallocationsandlessinternalconversions.ValuescanbepassedintointernalAQLfunctionswithoutcopyingthem.ThiscanleadtoreducedqueryexecutiontimesforqueriesthatuseC++-basedAQLfunctions.

"replace-or-with-in"and"use-index-for-sort"rules

Theserulesnowfireinsomeadditionalcases,whichallowssimplifyingindexlookupconditionsandremovingSortNodesfromexecutionplans.

Clusterstatemanagement

Thecluster'sinternalstateinformationisnowalsomanagedbyArangoDBinstances.Earlierversionsreliedonthirdpartysoftwarebeinginstalledforthestoringtheclusterstate.ThestateismanagedbydedicatedArangoDBinstances,whichcanbestartedinaspecialagencymode.Theseinstancescanoperateinadistributedfashion.Theywillautomaticallyelectoneofthemtobecometheirleader,beingresponsibileforstoringthestatechangessentfromserversinthecluster.Theotherinstanceswillautomaticallyfollowtheleaderandwilltransparentlystandinshoulditbecomeunavailable.Theagencyinstancesarealsoself-organizing:theywillcontinuouslyprobeeachotherandre-electleaders.Thecommunicationbetweentheagencyinstancesusetheconsensus-basedRAFTprotocol.

TheoperationsforstoringandretrievingclusterstateinformationarenowmuchlessexpensivefromanArangoDBclusternodeperspective,whichinturnallowsforfasterclusteroperationsthatneedtofetchorupdatetheoverallclusterstate.

_fromand _toattributesofedgesareupdatableandusableinindexes

InArangoDBpriorto3.0theattributes_fromand_toofedgesweretreatedspeciallywhenloadingorstoringedges.Thatspecialhandlingledtotheseattributesbeingnotasflexibleasregulardocumentattributes.Forexample,the_fromand_toattributevaluesofanexistingedgecouldnotbeupdatedoncetheedgewascreated.Nowthisispossibleviathesingle-documentAPIsandviaAQL.

Additionally,the_fromand_toattributescouldnotbeindexedinuser-definedindexes,e.g.tomakeeachcombinationof_fromand_tounique.Finally,as_fromand_toreferencedthelinkedcollectionsbycollectionidandnotbycollectionname,theirmeaningbecameunclearonceareferencedcollectionwasdropped.Thecollectionidstoredinedgesthenbecameunusable,andwhenaccessing

WhatsNewin3.0

554

suchedgethecollectionnamepartofitwasalwaystranslatedto_undefined.

InArangoDB3.0,the_fromand_tovaluesofedgesaresavedasregularstrings.Thisallowsusing_fromand_toinuser-definedindexes.Additionally,thisallowstoupdatethe_fromand_tovaluesofexistingedges.Furthermore,collectionsreferencedby_fromand_tovaluesmaybedroppedandre-createdlater.Any_fromand_tovaluesofedgespointingtosuchdroppedcollectionareunaffectedbythedropoperationnow.

UnifiedAPIsforCRUDoperations

TheCRUDAPIsfordocumentsandedgehavebeenunified.EdgescannowbeinsertedandmodifiedviathesameAPIsasdocuments._fromand_toattributevaluescanbepassedasregulardocumentattributesnow:

db.myedges.insert({_from:"myvertices/some",_to:"myvertices/other",...});

Passing_fromand_toseparatelyasitwasrequiredinearlierversionsisnotnecessaryanymorebutwillstillwork:

db.myedges.insert("myvertices/some","myvertices/other",{...});

TheCRUDoperationsnowalsosupportbatchvariantsthatworksonarraysofdocuments/edges,e.g.

db.myedges.insert([

{_from:"myvertices/some",_to:"myvertices/other",...},

{_from:"myvertices/who",_to:"myvertices/friend",...},

{_from:"myvertices/one",_to:"myvertices/two",...},

]);

ThebatchvariantsarealsoavailableinArangoDB'sHTTPAPI.Theycanbeusedtomoreefficientlycarryoutoperationswithmultipledocumentsthantheirsingle-documentequivalents,whichrequiredoneHTTPrequestperoperation.Withthebatchoperations,theHTTPrequest/responseoverheadcanbeamortizedacrossmultipleoperations.

Persistentindexes

ArangoDB3.0providesanexperimentalpersistentindexfeature.Persistentindexesstoretheindexvaluesondiskinsteadofin-memoryonly.Thismeanstheindexesdonotneedtoberebuiltin-memorywhenacollectionisloadedorreloaded,whichshouldimprovecollectionloadingtimes.

ThepersistentindexesinArangoDBarebasedontheRocksDBengine.Tocreateapersistentindexforacollection,createanindexoftype"rocksdb"asfollows:

db.mycollection.ensureIndex({type:"rocksdb",fields:["fieldname"]});

Thepersistentindexesaresorted,sotheyallowequalitylookupsandrangequeries.Notethatthefeatureisstillhighlyexperimentalandhassomeknowndeficiencies.Itwillbefinalizeduntilthereleaseofthe3.0stableversion.

UpgradedV8version

TheV8enginethatisusedinsideArangoDBtoexecuteJavaScriptcodehasbeenupgradedfromversion4.3.61to5.0.71.39.ThenewversionmakesseveralmoreES6featuresavailablebydefault,including

arrowfunctionscomputedpropertynamesrestparametersarraydestructuringnumericandobjectliterals

WhatsNewin3.0

555

WebAdminInterface

TheArangoDB3.0webinterfaceissignificantlyimproved.Itnowcomeswithamoreresponsivedesign,makingiteasiertouseondifferentdevices.Navigationandmenushavebeensimplified,andrelateditemshavebeenregroupedtostayclosertogetherandallowtighterworkflows.

TheAQLqueryeditorisnowmucheasiertouse.Multiplequeriescanbestartedandtrackedinparallel,whileresultsofearlierqueriesarestillpreserved.Queriesstillrunningcanbecanceleddirectlyfromtheeditor.TheAQLqueryeditornowallowstheusageofbindparameterstoo,andprovidesahelperforfindingcollectionnames,AQLfunctionnamesandkeywordsquickly.

Thewebinterfacenowkeepstrackofwhethertheserverisofflineandofwhichserver-sideoperationshavebeenstartedandarestillrunning.Itnowremainsusablewhilesuchlonger-runningoperationsareongoing.Italsokeepsmorestateaboutuser'schoices(e.g.windowssizes,whetherthetreeorthecodeviewwaslastusedinthedocumenteditor).

Clusterstatisticsarenowintegratedintothewebinterfaceaswell.Additionally,amenuitem"Helpus"hasbeenaddedtoeasilyprovidetheArangoDBteamfeedbackabouttheproduct.

Thefrontendmaynowbemountedbehindareverseproxyonadifferentpath.ForthistoworktheproxyshouldsendaX-Script-Nameheadercontainingthepath.

Abackendconfigurationforhaproxymightlooklikethis:

reqaddX-Script-Name:\/arangodb

Thefrontendwillrecognizethesubpathandproduceappropriatelinks.ArangoDBwillonlyacceptpathsfromtrustedfrontendproxies.Trustedproxiesmaybeaddedonstartup:

--frontend.proxy-request-checktrue--frontend.trusted-proxy192.168.1.117

--frontend.trusted-proxymaybeanyaddressornetmask.

Todisablethecheckandblindlyacceptanyx-script-nameset--frontend.proxy-request-checktofalse.

Foxximprovements

TheFoxxframeworkhasbeencompletelyrewrittenfor3.0withanew,simplerandmorefamiliarAPI.Themostnotablechangesare:

Legacymodefor2.8services

Stuckwitholdcode?Youcancontinueusingyour2.8-compatibleFoxxserviceswith3.0byadding"engines":{"arangodb":"^2.8.0"}(orsimilarversionrangesthatexclude3.0andup)totheservicemanifest.

Nomoreglobalvariablesandmagicalcomments

TheapplicationContextisnowmodule.context.Insteadofmagicalcommentsjustusethesummaryanddescriptionmethodstodocumentyourroutes.

RepositoryandModelhavebeenremoved

InsteadofrepositoriesjustuseArangoDBcollectionsdirectly.Forvalidationsimplyusethejoischemas(butwrappedinjoi.object())thatpreviouslylivedinsidethemodel.CollectionsandqueriesreturnplainJavaScriptobjects.

Controllershavebeenreplacedwithnestablerouters

Createrouterswithrequire('@arangodb/foxx/router')(),attachthemtoyourservicewithmodule.context.use(router).Becauseroutersarenolongermountedautomagically,youcanexportandimportthemlikeanyotherobject.Userouter.use('/path',subRouter)tonestroutersasdeeplyasyouwant.

Routescanbenamedandreversed

NomorememorizingURLs:addanametoyourroutelikerouter.get('/hello/:name',function(){...},'hello')andredirecttothefullURLwithres.redirect(req.resolve('hello',{name:'world'})).

WhatsNewin3.0

556

Simplerexpress-likemiddleware

Ifyoualreadyknowexpress,thisshouldbefamiliar.Here'sarequestloggerinthreelinesofcode:


varstart=Date.now();

try{next();}

finally{console.log(`${req.method}${req.url}${res.statusCode}${Date.now()-start}ms`);}

});

Sessionsandauthwithoutdependencies

Tomakeiteasiertogetstarted,thefunctionalitypreviouslyprovidedbythesimple-auth,oauth2,sessions-localandsessions-jwtserviceshavebeenmovedintoFoxxasthe@arangodb/foxx/auth,@arangodb/foxx/oauth2and@arangodb/foxx/sessionsmodules.

Logging

ArangoDB'sloggingisnowgroupedintotopics.Thelogverbosityandoutputfilescanbeadjustedperlogtopic.Forexample

--log.levelstartup=trace--log.levelqueries=trace--log.levelinfo

willlogmessagesconcerningstartupattracelevel,AQLqueriesattracelevelandeverythingelseatinfolevel.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.

Somerelevantlogtopicsavailablein3.0are:

collector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperations(includingmsync)queries:executedAQLqueries,slowqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads

Thisalsoallowsdirectinglogoutputtodifferentfilesbasedontopics.Forexample,tologallAQLqueriestoafile"queries.log"onecanusetheoptions:

--log.levelqueries=trace--log.outputqueries=file:///path/to/queries.log

ToadditionallylogHTTPrequesttoafilenamed"requests.log"addtheoptions:

--log.levelrequests=info--log.outputrequests=file:///path/to/requests.log

Buildsystem

ArangoDBnowusesthecross-platformbuildsystemCMakeforallitsbuilds.Previousversionsusedtwodifferentbuildsystems,makingdevelopmentandcontributionsharderthannecessary.Nowthebuildsystemisunified,andalltargets(Linux,Windows,MacOS)arebuiltfromthesamesetofbuildinstructions.

Documentation

Thedocumentationhasbeenenhancedandre-organizedtobemoreintuitive.

WhatsNewin3.0

557

AnewintroductionforbeginnersshouldbringyouuptospeedwithArangoDBinlessthananhour.Additionaltopicshavebeenintroducedandwillbeextendedwithupcomingreleases.

ThetopicsAQLandHTTPAPIarenowseparatedfromthemanualforbettersearchabilityandlessconfusion.Aversionswitchermakesiteasiertojumptotheversionofthedocsyouareinterestedin.

WhatsNewin3.0

558


Buildsystem

BuildingArangoDB3.0fromsourcenowrequiresCMake.

Thepre-3.0buildsystemusedaconfigure-basedapproach.ThestepstobuildArangoDB2.8fromsourcecodewere:

makesetup

./configure<options>

make

Thesestepswillnotworkanymore,asArangoDB3.0doesnotcomewithaconfigurescript.

Tobuild3.0onLinux,createaseparatebuilddirectoryfirst:

mkdir-pbuild

andthencreatetheinitialbuildscriptsonceusingCMake:

(cdbuild&&cmake<options>..)

Theabovecommandwillconfigurethebuildandcheckfortherequireddependencies.Ifeverythingworkswelltheactualbuildcanbestartedwith

(cdbuild&&make)

ThebinariesfortheArangoDBserverandallclienttoolswillthenbecreatedinsidethebuilddirectory.TostartArangoDBlocallyfromthebuilddirectory,use

build/bin/arangod<options>

Datafilesanddatafilenames

ArangoDB3.0usesanewVelocyPack-basedformatforstoringdatainWALlogfilesandcollectiondatafiles.ThefileformatisnotcompatiblewiththefilesusedinpriorversionsofArangoDB.ThatmeansdatafileswrittenbyArangoDB3.0cannotbeusedinearlierversionsandviceversa.

Thepatternforcollectiondirectorynameswaschangedin3.0toincludearandomidcomponentattheend.Thenewpatterniscollection-<id>-<random>,where<id>isthecollectionidand<random>isarandomnumber.PreviousversionsofArangoDBusedapatterncollection-<id>withouttherandomnumber.

UserManagementUnlikeArangoDB2.x,ArangoDB3.0usersarenowseparatedfromdatabases,andyoucangrantoneormoredatabasepermissionstoauser.

IfyouwanttomimicthebehaviorofArangoDB,youshouldnameyouruserslikeusername@dbname.

Usersthatcanaccessthe_systemdatabaseareallowedtomanageusersandpermissionsforalldatabases.


559

Edgesandedgesattributes

InArangoDBpriorto3.0theattributes_fromand_toofedgesweretreatedspeciallywhenloadingorstoringedges.Thatspecialhandlingledtotheseattributesbeingnotasflexibleasregulardocumentattributes.Forexample,the_fromand_toattributevaluesofanexistingedgecouldnotbeupdatedoncetheedgewascreated.Additionally,the_fromand_toattributescouldnotbeindexedinuser-definedindexes,e.g.tomakeeachcombinationof_fromand_tounique.Finally,as_fromand_toreferencedthelinkedcollectionsbycollectionidandnotbycollectionname,theirmeaningbecameunclearonceareferencedcollectionwasdropped.Thecollectionidstoredinedgesthenbecameunusable,andwhenaccessingsuchedgethecollectionnamepartofitwasalwaystranslatedto_undefined.

InArangoDB3.0,the_fromand_tovaluesofedgesaresavedasregularstrings.Thisallowsusing_fromand_toinuser-definedindexes.Additionallythisallowsupdatingthe_fromand_tovaluesofexistingedges.Furthermore,collectionsreferencedby_fromand_tovaluesmaybedroppedandre-createdlater.Any_fromand_tovaluesofedgespointingtosuchdroppedcollectionareunaffectedbythedropoperationnow.Alsonotethatrenamingthecollectionreferencedin_fromand_toinArangoDB2.8alsorelinkedtheedges.In3.0theedgesareNOTautomaticallyrelinkedtothenewcollectionanymore.

DocumentsDocuments(incontrasttoedges)cannotcontaintheattributes_fromor_toonthemainlevelinArangoDB3.0.Theseattributeswillbeautomaticallyremovedwhensavingdocuments(i.e.non-edges)._fromand_tocanbestillusedinsub-objectsinsidedocuments.

The_fromand_toattributeswillofcoursebepreservedandarestillrequiredwhensavingedges.

AQL

Edgeshandling

WhenupdatingorreplacingedgesviaAQL,anymodificationstothe_fromand_toattributesofedgeswereignoredbypreviousversionsofArangoDB,withoutsignalinganyerrors.Thiswasduetothe_fromand_toattributesbeingimmutableinearlierversionsofArangoDB.

From3.0on,the_fromand_toattributesofedgesaremutable,soanyAQLqueriesthatmodifythe_fromor_toattributevaluesofedgeswillattempttoactuallychangetheseattributes.Clientsshouldbeawareofthischangeandshouldreviewtheirqueriesthatmodifyedgestoruleoutunintendedside-effects.

Additionally,whencompletelyreplacingthedataofexistingedgesviatheAQLREPLACEoperation,itisnowrequiredtospecifyvaluesforthe_fromand_toattributes,asREPLACErequirestheentirenewdocumenttobespecified.Ifeither_fromor_toaremissingfromthereplacementdocument,anREPLACEoperationwillfail.

Graphfunctions

Inversion3.0allformergraphrelatedfunctionshavebeenremovedfromAQLtobereplacedbynativeAQLconstructs.Theseconstructsallowformorefine-grainedfilteringonseveralgraphlevels.AlsothisallowstheAQLoptimizertoautomaticallyimprovethesequeriesbyenhancingthemwithappropriateindexes.Wehavecreatedrecipestoupgradefrom2.8to3.0whenusingthesefunctions.

Thefunctions:

GRAPH_COMMON_NEIGHBORSGRAPH_COMMON_PROPERTIESGRAPH_DISTANCE_TOGRAPH_EDGESGRAPH_NEIGHBORSGRAPH_TRAVERSALGRAPH_TRAVERSAL_TREEGRAPH_SHORTEST_PATHGRAPH_PATHSGRAPH_VERTICES


560

arecoveredinMigratingGRAPH_*Functionsfrom2.8orearlierto3.0

GRAPH_ABSOLUTE_BETWEENNESSGRAPH_ABSOLUTE_CLOSENESSGRAPH_ABSOLUTE_ECCENTRICITYGRAPH_BETWEENNESSGRAPH_CLOSENESSGRAPH_DIAMETERGRAPH_ECCENTRICITYGRAPH_RADIUS

arecoveredinMigratingGRAPH_*Measurementsfrom2.8orearlierto3.0

EDGESNEIGHBORSPATHSTRAVERSALTRAVERSAL_TREE

arecoveredinMigratinganonymousgraphfunctionsfrom2.8orearlierto3.0

Typecastingfunctions

ThetypecastingappliedbytheTO_NUMBER()AQLfunctionhaschangedasfollows:

stringvaluesthatdonotcontainavalidnumericvaluearenowconvertedtothenumber0.InpreviousversionsofArangoDBsuchstringvalueswereconvertedtothevaluenull.arrayvalueswithmorethan1memberarenowconvertedtothenumber0.InpreviousversionsofArangoDBsucharrayswereconvertedtothevaluenull.objects/documentsarenowconvertedtothenumber0.InpreviousversionsofArangoDBobjects/documentswereconvertedtothevaluenull.

Additionally,theTO_STRING()AQLfunctionnowconvertsnullvaluesintoanemptystring("")insteadofthestring"null",whichismoreinlinewithLENGTH(null)returning0andnot4sincev2.6.

TheoutputofTO_STRING()hasalsochangedforarraysandobjectsasfollows:

arraysarenowconvertedintotheirJSON-stringifyequivalents,e.g.

[]isnowconvertedto[][1,2,3]isnowconvertedto[1,2,3]["test",1,2]isnowconvertedto["test",1,2]`

PreviousversionsofArangoDBconvertedarrayswithnomembersintotheemptystring,andnon-emptyarraysintoacomma-separatedlistofmembervalues,withoutthesurroundingangularbrackets.Additionally,stringarraymemberswerenotenclosedinquotesintheresultstring:

[]wasconvertedto``[1,2,3]wasconvertedto1,2,3["test",1,2]wasconvertedtotest,1,2`

objectsarenowconvertedtotheirJSON-stringifyequivalents,e.g.

{}isconvertedto{}{a:1,b:2}isconvertedto{"a":1,"b":2}{"test":"foobar"}isconvertedto{"test":"foobar"}

PreviousversionsofArangoDBalwaysconvertedobjectsintothestring[objectObject]

ThischangealsoaffectsotherpartsinAQLthatusedTO_STRING()toimplicitlycastoperandstostrings.ItalsoaffectstheAQLfunctionsCONCAT()andCONCAT_SEPARATOR()whichtreatedarrayvaluesdifferently.PreviousversionsofArangoDBautomaticallyflattenedarrayvaluesinthefirstlevelofthearray,e.g.CONCAT([1,2,3,[4,5,6]])produced1,2,3,4,5,6.Nowthiswillproduce[1,2,3,[4,5,6]].Toflattenarraymembersonthetoplevel,youcannowusethemoreexplicitCONCAT(FLATTEN([1,2,3,[4,5,6]],1)).


561

Arithmeticoperators

AsthearithmeticoperationsinAQLimplicitlyconverttheiroperandstonumericvaluesusingTO_NUMBER(),theircastingbehaviorhasalsochangedasdescribedabove.

Someexamplesofthechangedbehavior:

"foo"+1produces1now.Inpreviousversionsthisproducednull.[1,2]+1produces1.Inpreviousversionsthisproducednull.1+"foo"+1´produces2now.Inpreviousversionthisproduced1`.

Attributenamesandparameters

PreviousversionsofArangoDBhadsometroublewithattributenamesthatcontainedthedotsymbol(.).SomecodepartsinAQLusedthedotsymboltosplitanattributenameintosub-components,soanattributenameda.bwasnotcompletelydistinguishablefromanattributeawithasub-attributeb.Thisinconsistentbehaviorsometimesallowed"hacks"toworksuchaspassingsub-attributesinabindparameterasfollows:

FORdocINcollection

FILTERdoc.@name==1

RETURNdoc

Ifthebindparameter@namecontainedthedotsymbol(e.g.@bind=a.b,itwasunclearwhetherthisshouldtriggersub-attributeaccess(i.e.doc.a.b)oraaccesstoanattributewithexactlythespecifiedname(i.e.doc["a.b"]).

ArangoDB3.0nowhandlesattributenamescontainingthedotsymbolproperly,andsendingabindparameter@name=a.bwillnowalwaystriggeranaccesstotheattributedoc["a.b"],notthesub-attributebofaindoc.

Forusersthatusedthe"hack"ofpassingbindparameterscontainingdotsymboltoaccesssub-attributes,ArangoDB3.0allowsspecifyingtheattributenamepartsasanarrayofstrings,e.g.@name=["a","b"],whichwillberesolvedtothesub-attributeaccessdoc.a.bwhenthequeryisexecuted.

Keywords

LIKEisnowakeywordinAQL.UsingLIKEineithercaseasanattributeorcollectionnameinAQLqueriesnowrequiresquoting.

SHORTEST_PATHisnowakeywordinAQL.UsingSHORTEST_PATHineithercaseasanattributeorcollectionnameinAQLqueriesnowrequiresquoting.

Subqueries

Queriesthatcontainsubqueriesthatcontaindata-modificationoperationssuchasINSERT,UPDATE,REPLACE,UPSERTorREMOVEwillnowrefusetoexecuteifthecollectionaffectedbythesubquery'sdata-modificationoperationisread-accessedinanouterscopeofthequery.

Forexample,thefollowingquerywillrefusetoexecuteasthecollectionmyCollectionismodifiedinthesubquerybutalsoread-accessedintheouterscope:

FORdocINmyCollection

LETchanges=(

FORwhatINmyCollection

FILTERwhat.value==1

REMOVEwhatINmyCollection

)

RETURNdoc

Itisstillpossibletowritetocollectionsfromwhichdataisreadinthesamequery,e.g.

FORdocINmyCollection

FILTERdoc.value==1

REMOVEdocINmyCollection


562

andtomodifydataindifferentcollectionviasubqueries.

Otherchanges

TheAQLoptimizerrule"merge-traversal-filter"thatalreadyexistedin3.0wasrenamedto"optimize-traversals".ThisshouldbeofnorelevancetoclientapplicationsexceptiftheyprogramaticallylookforappliedoptimizerrulesintheexplainoutofAQLqueries.

TheorderofresultscreatedbytheAQLfunctionsVALUES()andATTRIBUTES()wasneverguaranteedanditonlyhadthe"correct"orderingbyaccidentwheniteratingoverobjectsthatwerenotloadedfromthedatabase.Assomeofthefunctioninternalshavechanged,the"correct"orderingwillnotappearanymore,andstillnoresultorderisguaranteedbythesefunctionsunlessthesortparameterisspecified(fortheATTRIBUTES()function).

UpgradedV8version

TheV8enginethatisusedinsideArangoDBtoexecuteJavaScriptcodehasbeenupgradedfromversion4.3.61to5.0.71.39.Thenewversionshouldbemostlycompatibletotheoldversion,buttheremaybesubtledifferences,includingchangesoferrormessagetextsthrownbytheengine.Furthermore,someV8startupparametershavechangedtheirmeaningorhavebeenremovedinthenewversion.ThisisonlyrelevantwhenArangoDBorArangoShellarestartedwithacustomvalueforthe--javascript.v8-optionsstartupoption.

Amongothers,thefollowingV8optionschangeinthenewversionofArangoDB:

--es_staging:in2.8ithadthemeaningenableallcompletedharmonyfeatures,in3.0theoptionmeansenabletest-worthyharmonyfeatures(forinternaluseonly)

--strong_this:thisoptionwasn'tpresentin2.8.In3.0itmeansdon'tallow'this'toescapefromconstructorsanddefaultstotrue.

--harmony_regexps:thisoptionsmeansenable"harmonyregularexpressionextensions"andchangesitsdefaultvaluefromfalsetotrue

--harmony_proxies:thisoptionsmeansenable"harmonyproxies"andchangesitsdefaultvaluefromfalsetotrue

--harmony_reflect:thisoptionsmeansenable"harmonyReflectAPI"andchangesitsdefaultvaluefromfalsetotrue

--harmony_sloppy:thisoptionsmeansenable"harmonyfeaturesinsloppymode"andchangesitsdefaultvaluefromfalsetotrue

--harmony_tostring:thisoptionsmeansenable"harmonytoString"andchangesitsdefaultvaluefromfalsetotrue

--harmony_unicode_regexps:thisoptionsmeansenable"harmonyunicoderegexps"andchangesitsdefaultvaluefromfalsetotrue

--harmony_arrays,--harmony_array_includes,--harmony_computed_property_names,--harmony_arrow_functions,--harmony_rest_parameters,--harmony_classes,--harmony_object_literals,--harmony_numeric_literals,--harmony_unicode:theseoptionhavebeenremovedinV85.

AsaconsequenceoftheupgradetoV8version5,theimplementationoftheJavaScriptBufferobjecthadtobechanged.JavaScriptBufferobjectsinArangoDBnowalwaysstoretheirdataontheheap.ThereisnosharedpoolforsmallBuffervalues,andnopointingintoexistingBufferdatawhenextractingslices.ThischangemayincreasethecostofcreatingBufferswithshortcontentsorwhenpeekingintoexistingBuffers,butwasrequiredforsafermemorymanagementandtopreventleaks.

JavaScriptAPIchanges

ThefollowingincompatiblechangeshavebeenmadetotheJavaScriptAPIinArangoDB3.0:

Foxx

TheFoxxframeworkhasbeencompletelyrewrittenfor3.0withanew,simplerandmorefamiliarAPI.TomakeFoxxservicesdevelopedfor2.8orearlierArangoDBversionsrunin3.0,theservice'smanifestfileneedstobeedited.

ToenablethelegacymodeforaFoxxservice,add"engines":{"arangodb":"^2.8.0"}(orsimilarversionrangesthatexclude3.0andup)totheservicemanifestfile(named"manifest.json",locatedintheservice'sbasedirectory).


563

Require

ModulesshippedwithArangoDBcannowberequiredusingthepattern@arangodb/<module>insteadoforg/arangodb/<module>,e.g.

varcluster=require("@arangodb/cluster");

Theoldformatcanstillbeusedforcompatibility:

varcluster=require("org/arangodb/cluster");

ArangoDBpriortoversion3.0allowedatransparentuseofCoffeeScriptsourcefileswiththerequire()function.FileswithafilenameextensionofcoffeewereautomaticallysentthroughaCoffeeScriptparserandtranspiledintoJavaScripton-the-fly.ThissupportisgonewithArangoDB3.0.TorunanyCoffeeScriptsourcefiles,theymustbeconvertedtoJavaScriptbytheclientapplication.

Responseobject

The@arangodb/requestresponseobjectnowstorestheparsedJSONresponsebodyinapropertyjsoninsteadofbodywhentherequestwasmadeusingthejsonoption.Thebodyinsteadcontainstheresponsebodyasastring.

EdgesAPI

Whencompletelyreplacinganedgeviaacollection'sreplace()functionthereplacingedgedatanowneedstocontainthe_fromand_toattributesforthenewedge.PreviousversionsofArangoDBdidnotrequiretheedgedatatocontain_fromand_toattributeswhenreplacinganedge,since_fromand_tovalueswereimmutableforexistingedges.

Forexample,thefollowingcallworkedinArangoDB2.8butwillfailin3.0:

db.edgeCollection.replace("myKey",{value:"test"});

TomakethisworkinArangoDB3.0,_fromand_toneedtobeaddedtothereplacementdata:

db.edgeCollection.replace("myKey",{_from:"myVertexCollection/1",_to:"myVertexCollection/2",value:"test"});

Notethatthisonlyaffectsthereplace()functionbutnotupdate(),whichwillonlyupdatethespecifiedattributesoftheedgeandleaveallothersintact.

Additionally,thefunctionsedges(),outEdges()andinEdges()withanarrayofedgeidswillnowmaketheedgeidsuniquebeforereturningtheconnectededges.Thisisprobablydesiredanyway,asresultswillbereturnedonlyonceperdistinctinputedgeid.However,itmaybreakclientapplicationsthatrelyontheoldbehavior.

DatabasesAPI

The_listDatabases()functionofthedbobjecthasbeenrenamedto_databases(),makingitconsistentwiththe_collections()function.Alsothe_listEndpoints()functionhasbeenrenamedto_endpoints().

CollectionAPI

Examplematching

ThecollectionfunctionbyExampleHash()andbyExampleSkiplist()havebeenremovedin3.0.Theirfunctionalityisprovidedbycollection'sbyExample()function,whichwillautomaticallyuseasuitableindexifpresent.

ThecollectionfunctionbyConditionSkiplist()hasbeenremovedin3.0.ThesamefunctionalitycanbeachievedbyissuinganAQLquerywiththetargetcondition,whichwillautomaticallyuseasuitableindexifpresent.

Revisionidhandling


564

Theexists()methodofacollectionnowthrowsanexceptionwhenthespecifieddocumentexistsbutitsrevisioniddoesnotmatchtherevisionidspecified.PreviousversionsofArangoDBsimplyreturnedfalseifeithernodocumentexistedwiththespecifiedkeyorwhentherevisioniddidnotmatch.Itwasthereforeimpossibletodistinguishthesetwocasesfromthereturnvaluealone.3.0correctsthis.Additionally,exists()inpreviousversionsalwaysreturnedabooleanifonlythedocumentkeywasgiven.3.0nowreturnsthedocument'smeta-data,whichincludesthedocument'scurrentrevisionid.

GiventhereisadocumentwithkeytestincollectionmyCollection,thenthebehaviorof3.0isasfollows:

/*testifdocumentexists.thisreturnedtruein2.8*/

db.myCollection.exists("test");

{

"_key":"test",

"_id":"myCollection/test",

"_rev":"9758059"

}

/*testifdocumentexists.thisreturnedtruein2.8*/

db.myCollection.exists({_key:"test"});

{

"_key":"test",


"_rev":"9758059"

}

/*testifdocumentexists.thisalsoreturnedfalsein2.8*/

db.myCollection.exists("foo");

false

/*testifdocumentwithagivenrevisionidexists.thisreturnedtruein2.8*/

db.myCollection.exists({_key:"test",_rev:"9758059"});

{

"_key":"test",


"_rev":"9758059"

}

/*testifdocumentwithagivenrevisionidexists.thisreturnedfalsein2.8*/

db.myCollection.exists({_key:"test",_rev:"1234"});

JavaScriptexception:ArangoError1200:conflict

Capconstraints

Thecapconstraintsfeaturehasbeenremoved.Thischangehasledtotheremovalofthecollectionoperationsfirst()andlast(),whichwereinternallybasedondatafromcapconstraints.

AscapconstraintshavebeenremovedinArangoDB3.0itisnotpossibletocreateanindexoftype"cap"withacollection'sensureIndex()function.ThededicatedfunctionensureCapConstraint()hasalsobeenremovedfromthecollectionAPI.

GraphBlueprintsJSModule

Thedeprecatedmodulegraph-blueprintshasbeendeleted.Allit'sfeaturesarecoveredbythegeneral-graphmodule.

GeneralGraphFluentAQLinterface

ThefluentinterfacehasbeenremovedfromArangoDB.It'sfeatureswerecompletelyoverlappingwith"aqb"whichcomespreinstalledaswell.PleaseswitchtoAQBinstead.

UndocumentedAPIs

TheundocumentedfunctionsBY_EXAMPLE_HASH()andBY_EXAMPLE_SKIPLIST(),BY_CONDITION_SKIPLIST,CPP_NEIGHBORSandCPP_SHORTEST_PATHhavebeenremoved.ThesefunctionswerealwayshiddenandnotintendedtobepartofthepublicJavaScriptAPIforcollections.

HTTPAPIchanges


565

https://github.com/arangodb/aqbjs

CRUDoperations

ThefollowingincompatiblechangeshavebeenmadetotheHTTPAPIinArangoDB3.0:

General

TheHTTPinsertoperationsforsingledocumentsandedges(POST/_api/document)donotsupporttheURLparameter"createCollection"anymore.InpreviousversionsofArangoDBthisparametercouldbeusedtoautomaticallycreateacollectionuponinsertionofthefirstdocument.ItisnowrequiredthatthetargetcollectionalreadyexistswhenusingthisAPI,otherwiseitwillreturnanHTTP404error.ThesameistruefortheimportAPIatPOST/_api/import.

CollectionscanstillbecreatedeasilyviaaseparatecalltoPOST/_api/collectionasbefore.

The"location"HTTPheaderreturnedbyArangoDBwheninsertinganewdocumentoredgenowalwayscontainsthedatabasename.ThiswasalsothedefaultbehaviorinpreviousversionsofArangoDB,butitcouldbeoverriddenbyclientssendingtheHTTPheaderx-arango-version:1.4intherequest.ClientscancontinuetosendthisheadertoArangoDB3.0,buttheheaderwillnotinfluencethelocationresponseheadersproducedbyArangoDB3.0anymore.

AdditionallytheCRUDoperationsAPIsdonotreturnanattribute"error"intheresponsebodywithanattributevalueof"false"incaseanoperationsucceeded.

Revisionidhandling

Theoperationsforupdating,replacingandremovingdocumentscanoptionallychecktherevisionnumberofthedocumenttobeupdated,replacedorremovedsothecallercanensuretheoperationworksonaspecificversionofthedocumentandtherearenolostupdates.

PreviousversionsofArangoDBallowedpassingtherevisionidofthepreviousdocumenteitherintheHTTPheaderIf-MatchorintheURLparameterrev.Forexample,removingadocumentwithaspecificrevisionidcouldbeachievedasfollows:

curl-XDELETE\

"http://127.0.0.1:8529/_api/document/myCollection/myKey?rev=123"

ArangoDB3.0doesnotsupportpassingtherevisionidviathe"rev"URLparameteranymore.InsteadthepreviousrevisionidmustbepassedintheHTTPheaderIf-Match,e.g.

curl-XDELETE\

--header"If-Match:'123'"\

"http://127.0.0.1:8529/_api/document/myCollection/myKey"

TheURLparameter"policy"wasalsousableinpreviousversionsofArangoDBtocontrolrevisionhandling.Usingitwasredundanttospecifyingtheexpectedrevisionidviathe"rev"parameteror"If-Match"HTTPheaderandthereforesupportforthe"policy"parameterwasremovedin3.0.

Inordertocheckforapreviousrevisionidwhenupdating,replacingorremovingdocumentspleaseusetheIf-MatchHTTPheaderasdescribedabove.WhennorevisioncheckifrequiredtheHTTPheadercanbeomitted,andtheoperationswillworkonthecurrentrevisionofthedocument,regardlessofitsrevisionid.

AlldocumentsAPI

TheHTTPAPIforretrievingtheids,keysorURLsofalldocumentsfromacollectionwaspreviouslylocatedatGET/_api/document?collection=....ThisAPIwasmovedtoPUT/_api/simple/all-keysandisnowexecutedasanAQLquery.ThenameofthecollectionmustnowbepassedintheHTTPrequestbodyinsteadofintherequestURL.Thesameistrueforthe"type"parameter,whichcontrolsthetypeoftheresulttobecreated.

CallstothepreviousAPIcanbetranslatedasfollows:

old:GET/_api/document?collection=<collection>&type=<type>withoutHTTPrequestbody3.0:PUT/_api/simple/all-keyswithHTTPrequestbody{"collection":"<collection>","type":"id"}

TheresultformatofthisAPIhasalsochangedslightly.InpreviousversionscallstotheAPIreturnedaJSONobjectwithadocumentsattribute.AsthefunctionalityisbasedonAQLinternallyin3.0,theAPInowreturnsaJSONobjectwitharesultattribute.


566

EdgesAPI

CRUDoperations

TheAPIfordocumentsandedgeshavebeenunifiedinArangoDB3.0.TheCRUDoperationsfordocumentsandedgesarenowhandledbythesameendpointat/_api/document.ForCRUDoperationsthereisnodistinctionanymorebetweendocumentsandedgesAPI-wise.

ThatmeansCRUDoperationsconcerningedgesneedtobesenttotheHTTPendpoint/_api/documentinsteadof/_api/edge.Sendingrequeststo/_api/edgewillresultinanHTTP404errorin3.0.Thefollowingmethodsareavailableat/_api/documentfordocumentsandedge:

HTTPPOST:insertnewdocumentoredgeHTTPGET:fetchanexistingdocumentoredgeHTTPPUT:replaceanexistingdocumentoredgeHTTPPATCH:partiallyupdateanexistingdocumentoredgeHTTPDELETE:removeanexistingdocumentoredge

WhencompletelyreplacinganedgeviaHTTPPUTpleasenotethatthereplacingedgedatanowneedstocontainthe_fromand_toattributesfortheedge.PreviousversionsofArangoDBdidnotrequiresending_fromand_towhenreplacingedges,as_fromand_tovalueswereimmutableforexistingedges.

The_fromand_toattributesofedgesnowalsoneedtobepresentinsidetheedgesobjectssenttotheserver:

curl-XPOST\

--data'{"value":1,"_from":"myVertexCollection/1","_to":"myVertexCollection/2"}'\

"http://127.0.0.1:8529/_api/document?collection=myEdgeCollection"

PreviousversionsofArangoDBrequiredthe_fromand_toattributesofedgesbesentseparatelyinURLparameterfromandto:

curl-XPOST\

--data'{"value":1}'\

"http://127.0.0.1:8529/_api/edge?collection=e&from=myVertexCollection/1&to=myVertexCollection/2"

Queryingconnectededges

TheRESTAPIforqueryingconnectededgesatGET/_api/edges/<collection>willnowmaketheedgeidsuniquebeforereturningtheconnectededges.Thisisprobablydesiredanywayasresultswillnowbereturnedonlyonceperdistinctinputedgeid.However,itmaybreakclientapplicationsthatrelyontheoldbehavior.

GraphAPI

Somedata-modificationoperationsinthenamedgraphsAPIat/_api/gharialnowreturneitherHTTP202(Accepted)orHTTP201(Created)iftheoperationsucceeds.WhichstatuscodeisreturneddependsonthewaitForSyncattributeoftheaffectedcollection.InpreviousversionssomeoftheseoperationsreturnHTTP200regardlessofthewaitForSyncvalue.

ThedeprecatedgraphAPI/_api/graphhasbeenremoved.Allit'sfeaturescanbereplacedusing/_api/gharialandAQLinstead.

SimplequeriesAPI

TheRESTroutesPUT/_api/simple/firstand/_api/simple/lasthavebeenremovedentirely.TheseAPIswereresponsibleforreturningthefirst-insertedandlast-inserteddocumentsinacollection.Thisfeaturewasbuiltoncapconstraintsinternally,whichhavebeenremovedin3.0.

Callingoneoftheseendpointsin3.0willresultinanHTTP404error.

IndexesAPI


567

Itisnotsupportedin3.0tocreateanindexwithtypecap(capconstraint)in3.0asthecapconstraintsfeaturehasbeeremoved.CallingtheindexcreationendpointHTTPAPIPOST/_api/index?collection=...withanindextypecapwillthereforeresultinanHTTP400error.

LogentriesAPI

TheRESTrouteHTTPGET/_admin/logisnowaccessiblefromwithinalldatabases.InpreviousversionsofArangoDB,thisroutewasaccessiblefromwithinthe_systemdatabaseonly,andanHTTP403(Forbidden)wasthrownbytheserverforanyaccessfromwithinanotherdatabase.

FiguresAPI

TheRESTrouteHTTPGET/_api/collection/<collection>/figureswillnotreturnthefollowingresultattributesastheybecamemeaninglessin3.0:

shapefiles.countshapes.fileSizeshapes.countshapes.sizeattributes.countattributes.size

DatabasesandCollectionsAPIs

WhencreatingadatabaseviatheAPIPOST/_api/database,ArangoDBwillnowalwaysreturntheHTTPstatuscode202(created)iftheoperationsucceeds.PreviousversionsofArangoDBreturnedHTTP202aswell,butthisbehaviorwaschangablebysendinganHTTPheaderx-arango-version:1.4.Whensendingthisheader,previousversionsofArangoDBreturnedanHTTPstatuscode200(ok).ClientscanstillsendthisheadertoArangoDB3.0butthiswillnotinfluencetheHTTPstatuscodeproducedbyArangoDB.

The"location"headerproducedbyArangoDB3.0willnowalwayscontainthedatabasename.ThiswasalsothedefaultinpreviousversionsofArangoDB,butthebehaviorcouldbeoverriddenbysendingtheHTTPheaderx-arango-version:1.4.Clientscanstillsendtheheader,butthiswillnotmakethedatabasenameinthe"location"responseheaderdisappear.

TheresultformatforqueryingallcollectionsviatheAPIGET/_api/collectionhasbeenchanged.

PreviousversionsofArangoDBreturnedanobjectwithanattributenamedcollectionsandanattributenamednames.Bothcontainedallavailablecollections,butcollectionscontainedthecollectionsasanarray,andnamescontainedthecollectionsagain,containedinanobjectinwhichtheattributenameswerethecollectionnames,e.g.

{

"collections":[

{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},

{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},

...

],

"names":{

"test":{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},

"something":{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},

...

}

}

Thisresultstructurewasredundant,andthereforehasbeensimplifiedtojust

{

"result":[

{"id":"5874437","name":"test","isSystem":false,"status":3,"type":2},

{"id":"17343237","name":"something","isSystem":false,"status":3,"type":2},

...

]

}

inArangoDB3.0.


568

ReplicationAPIs

TheURLparameter"failOnUnknown"wasremovedfromtheRESTAPIGET/_api/replication/dump.Thisparametercontrolledwhetherdumpingorreplicatingedgesshouldfailifoneofthevertexcollectionslinkedintheedge's_fromor_toattributeswasnotpresentanymore.Inthiscasethe_fromand_tovaluescouldnotbetranslatedintomeaningfulidsanymore.

Thereweretwowaysforhandlingthis:

settingfailOnUnknowntotruecausedtheHTTPrequesttofail,leavingerrorhandlingtotheusersettingfailOnUnknowntofalsecausedtheHTTPrequesttocontinue,translatingthecollectionnamepartinthe_fromor_tovalueto_unknown.

InArangoDB3.0thisparameterisobsolete,as_fromand_toarestoredasself-containedstringvaluesallthetime,sotheycannotgetinvalidwhenreferencedcollectionsaredropped.

TheresultformatoftheAPIGET/_api/replication/logger-followhaschangedslightlyinthefollowingaspects:

documentsandedgesarereportedinthesameway.Thetypefordocumentinsertions/updatesandedgeinsertions/updatesisnowalways2300.PreviousversionsofArangoDBreturnedatypevalueof2300fordocumentsand2301foredges.recordsaboutinsertions,updatesorremovalsofdocumentsandedgesdonothavethekeyandrevattributesonthetop-levelanymore.Instead,keyandrevcanbeaccessedbypeekingintothe_keyand_revattributesofthedatasub-attributesofthechangerecord.

Thesameistrueforthecollection-specificchangesAPIGET/_api/replication/dump.

UsermanagementAPIs

TheRESTAPIendpointPOST/_api/userforaddingnewusersnowrequirestherequesttocontainaJSONobjectwithanattributenameduser,containingthenameoftheusertobecreated.PreviousversionsofArangoDBalsocheckedthisattribute,butadditionallylookedforanattributeusernameiftheuserattributedidnotexist.

UndocumentedAPIs

ThefollowingundocumentedHTTPRESTendpointshavebeenremovedfromArangoDB'sRESTAPI:

/_open/cerberusand/_system/cerberus:theseendpointswereintendedforsomeArangoDB-internalapplicationsonlyPUT/_api/simple/by-example-hash,PUT/_api/simple/by-example-skiplistandPUT/_api/simple/by-condition-skiplist:thesemethodsweredocumentedinearlyversionsofArangoDBbuthavebeenmarkedasnotintendedtobecalledbyenduserssinceArangoDBversion2.3.ThesemethodsshouldnothavebeenpartofanyArangoDBmanualsinceversion2.4./_api/structure:anolderunfinishedandunpromotedAPIfordataformatandtypechecks,supersededbyFoxxapplications.

AdministrationAPIs

/_admin/shutdownnowneedstobecalledwiththeHTTPDELETEmethod

HandlingofCORSrequests

ItcannowbecontrolledindetailforwhichoriginhostsCORS(Cross-originresourcesharing)requestswithcredentialswillbeallowed.ArangoDB3.0providesthestartupoption--http.trusted-originthatcanbeusedtospecifyoneormanyoriginsfromwhichCORSrequestsaretreatedas"trustworthy".

Theoptioncanbespecifiedmultipletimes,oncepertrustedorigin,e.g.

--http.trusted-originhttp://127.0.0.1:8529--http.trusted-originhttps://127.0.0.1:8599

ThiswillmaketheArangoDBserverrespondtoCORSrequestsfromtheseoriginswithanAccess-Control-Allow-CredentialsHTTPheaderwithavalueoftrue.WebbrowserscaninspectthisheaderandcanallowpassingArangoDBwebinterfacecredentials(ifstoredinthebrowser)totherequestingsite.ArangoDBwillnotforwardorprovideanycredentials.


569

SettingthisoptionisonlyrequiredifapplicationsonotherhostsneedtoaccesstheArangoDBwebinterfaceorotherHTTPRESTAPIsfromawebbrowserwiththesamecredentialsthattheuserhasenteredwhenloggingintothewebinterface.WhenawebbrowserfindstheAccess-Control-Allow-CredentialsHTTPresponseheader,itmayforwardthecredentialsenteredintothebrowserfortheArangoDBwebinterfacelogintotheothersite.

Thisisapotentialsecurityissue,sotherearenotrustedoriginsbydefault.Itmayberequiredtosetsometrustedoriginsifyou'replanningtoissueAJAXrequeststoArangoDBfromothersitesfromthebrowser,withthecredentialsenteredduringtheArangoDBinterfacelogin(i.e.singlesign-on).Ifsuchfunctionalityisnotused,theoptionshouldnotbeset.

Tospecifyatrustedorigin,specifytheoptiononcepertrustedoriginasshownabove.NotethatthetrustedoriginvaluesspecifiedinthisoptionwillbecomparedbytewisewiththeOriginHTTPheadervaluesentbyclients,andonlyexactmatcheswillpass.

ThereisalsothewildcardallforenablingCORSaccessfromalloriginsinatestordevelopmentsetup:

--http.trusted-originall

SettingthisoptionwillleadtotheArangoDBserverrespondingwithanAccess-Control-Allow-Credentials:trueHTTPheadertoallincomingCORSrequests.

Command-lineoptionsQuiteafewstartupoptionsinArangoDB2weredoublenegations(like--server.disable-authenticationfalse).InArangoDB3thesearenowexpressedaspositives(e.g.--server.authentication).AlsotheoptionsbetweentheArangoDBserveranditsclienttoolshavebeingunified.Forexample,theloggeroptionsarenowthesamefortheserverandtheclienttools.Additionallymanyoptionshavebeenmovedintomoreappropriatetopicsections.

Renamedoptions

Thefollowingoptionshavebeenavailablebefore3.0andhavechangedtheirnamein3.0:

--server.disable-authenticationwasrenamedto--server.authentication.Notethatthemeaningoftheoption--server.authenticationistheoppositeoftheprevious--server.disable-authentication.--server.disable-authentication-unix-socketswasrenamedto--server.authentication-unix-sockets.Notethatthemeaningoftheoption--server.authentication-unix-socketsistheoppositeoftheprevious--server.disable-authentication-unix-sockets.--server.authenticate-system-onlywasrenamedto--server.authentication-system-only.Themeaningoftheoptioninunchanged.--server.disable-statisticswasrenamedto--server.statistics.Notethatthemeaningoftheoption--server.statisticsistheoppositeoftheprevious--server.disable-statistics.--server.cafilewasrenamedto--ssl.cafile.Themeaningoftheoptionisunchanged.--server.keyfilewasrenamedto--ssl.keyfile.Themeaningoftheoptionisunchanged.--server.ssl-cachewasrenamedto--ssl.session-cache.Themeaningoftheoptionisunchanged.--server.ssl-cipher-listwasrenamedto--ssl.cipher-list.Themeaningoftheoptionisunchanged.--server.ssl-optionswasrenamedto--ssl.options.Themeaningoftheoptionisunchanged.--server.ssl-protocolwasrenamedto--ssl.protocol.Themeaningoftheoptionisunchanged.--server.backlog-sizewasrenamedto--tcp.backlog-size.Themeaningoftheoptionisunchanged.--server.reuse-addresswasrenamedto--tcp.reuse-address.Themeaningoftheoptionisunchanged.--server.disable-replication-applierwasrenamedto--database.replication-applier.Themeaningoftheoption--database.replication-applieristheoppositeoftheprevious--server.disable-replication-applier.--server.allow-method-overridewasrenamedto--http.allow-method-override.Themeaningoftheoptionisunchanged.--server.hide-product-headerwasrenamedto--http.hide-product-header.Themeaningoftheoptionisunchanged.--server.keep-alive-timeoutwasrenamedto--http.keep-alive-timeout.Themeaningoftheoptionisunchanged.--server.foxx-queueswasrenamedto--foxx.queues.Themeaningoftheoptionisunchanged.--server.foxx-queues-poll-intervalwasrenamedto--foxx.queues-poll-interval.Themeaningoftheoptionisunchanged.--no-serverwasrenamedto--server.rest-server.Notethatthemeaningoftheoption--server.rest-serveristheoppositeoftheprevious--no-server.--database.query-cache-modewasrenamedto--query.cache-mode.Themeaningoftheoptionisunchanged.--database.query-cache-max-resultswasrenamedto--query.cache-entries.Themeaningoftheoptionisunchanged.


570

--database.disable-query-trackingwasrenamedto--query.tracking.Themeaningoftheoption--query.trackingistheoppositeoftheprevious--database.disable-query-tracking.--log.ttywasrenamedto--log.foreground-tty.Themeaningoftheoptionisunchanged.--upgradehasbeenrenamedto--database.auto-upgrade.Incontrastto2.8thisoptionnowrequiresabooleanparameter.Toactuallyperformanautomaticdatabaseupgradeatstartupuse--database.auto-upgradetrue.Tonotperformit,use--database.auto-upgradefalse.--check-versionhasbeenrenamedto--database.check-version.--temp-pathhasbeenrenamedto--temp.path.

Logverbosity,topicsandoutputfiles

Loggingnowsupportslogtopics.Youcancontrolthesebyspecifyingalogtopicinfrontofalogleveloranoutput.Forexample

--log.levelstartup=trace--log.levelinfo

willlogmessagesconcerningstartupattracelevel,everythingelseatinfolevel.--log.levelcanbespecifiedmultipletimesatstartup,forasmanytopicsasneeded.

Somerelevantlogtopicsavailablein3.0are:

collector:informationabouttheWALcollector'sstatecompactor:informationaboutthecollectiondatafilecompactordatafiles:datafile-relatedoperationsmmap:informationaboutmemory-mappingoperationsperformance:someperformance-relatedinformationqueries:executedAQLqueriesreplication:replication-relatedinforequests:HTTPrequestsstartup:informationaboutserverstartupandshutdownthreads:informationaboutthreads

Thenewlogoption--log.output<definition>allowsdirectingtheglobalorper-topiclogoutputtodifferentoutputs.Theoutputdefinition""canbeoneof

"-"forstdin"+"forstderr"syslog://""syslog:///""file://"

Theoptioncanbespecifiedmultipletimesinordertoconfiguretheoutputfordifferentlogtopics.Tosetupaper-topicoutputconfiguration,use--log.output<topic>=<definition>,e.g.

queries=file://queries.txt

logsallqueriestothefile"queries.txt".

Theoldoption--log.fileisstillavailablein3.0forconveniencereasons.In3.0itisashortcutforthemoregeneraloption--log.outputfile://filename.

Theoldoption--log.requests-fileisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.outputrequests=file://....

Theoldoption--log.performanceisstillavailablein3.0.Itisnowashortcutforthemoregeneraloption--log.levelperformance=trace.

Removedoptionsforlogging

Theoptions--log.content-filterand--log.source-filterhavebeenremoved.TheyhavemostbeenusedduringArangoDB'sinternaldevelopment.


571

Thesyslog-relatedoptions--log.applicationand--log.facilityhavebeenremoved.Theyaresupersededbythemoregeneral--log.outputoptionwhichcanalsohandlesyslogtargets.

Removedotheroptions

Theoption--server.default-api-compatibilitywaspresentinearlierversionofArangoDBtocontrolvariousaspectsoftheserverbehavior,e.g.HTTPreturncodesortheformatofHTTP"location"headers.ClientapplicationscouldsendanHTTPheader"x-arango-version"withaversionnumbertorequesttheserverbehaviorofacertainArangoDBversion.

Thisoptionwasonlyhonoredinahandfulofcases(describedabove)andwasremovedin3.0becausethechangesinserverbehaviorcontrolledbythisoptionwerechangedevenbeforeArangoDB2.0.Thisshouldhaveleftenoughtimeforclientapplicationstoadapttothenewbehavior,makingtheoptionsuperfluousin3.0.

Threadoptions

Theoptions--server.threadsand--scheduler.threadsnowhaveadefaultvalueof0.When--server.threadsissetto0onstartup,thesuitablenumberofthreadswillbedeterminedbyArangoDBbyaskingtheOSforthenumberofavailableCPUsandusingthatasabaseline.IfthenumberofCPUsislowerthan4,ArangoDBwillstillstart4dispatcherthreads.When--scheduler.threadsissetto0,thenArangoDBwillautomaticallydeterminethenumberofschedulerthreadstostart.Thiswillnormallycreate2schedulerthreads.

Iftheexactnumberofthreadsneedstobesetbytheadmin,thenitisstillpossibletoset--server.threadsand--scheduler.threadstonon-zerovalues.ArangoDBwillusethesevaluesandstartthatmanythreads(notethatsomethreadsmaybecreatedlazilysotheymaynotbepresentdirectlyafterstartup).

ThenumberofV8JavaScriptcontextstobecreated(--javascript.v8-contexts)nowhasadefaultvalueof0too,meaningthatArangoDBwillcreateasmanyV8contextsastherewillbedispatcherthreads(controlledbythe--server.threadsoption).Settingthisoptiontoanon-zerovaluewillcreateexactlyasmanyV8contextsasspecified.

Settingtheseoptionsexplicitlytonon-zerovaluesmaybebeneficialinenvironmentsthathavefewresources(processingtime,maximumthreadcount,availablememory).

AuthenticationThedefaultvaluefor--server.authenticationisnowtrueintheconfigurationfilesshippedwithArangoDB.Thismeanstheserverwillbestartedwithauthenticationenabledbydefault,requiringallclientconnectionstoprovideauthenticationdatawhenconnectingtoArangoDBAPIs.PreviousArangoDBversionsusedthesetting--server.disable-authenticationtrue,effectivelydisablingauthenticationbydefault.

Thedefaultvaluefor--server.authentication-system-onlyisnowtrueinArangoDB.ThatmeansthatFoxxapplicationsrunninginArangoDBwillbepublicaccessible(atleasttheywillnotuseArangoDB'sbuiltinauthenticationmechanism).OnlyrequeststoArangoDBAPIsatURLpathprefixes/_api/and/_adminwillrequireauthentication.Tochangethat,andusethebuiltinauthenticationmechanismforFoxxapplicationstoo,set--server.authentication-system-onlytofalse,andmakesuretohavetheoption--server.authenticationsettotrueaswell.

Thoughenablingtheauthenticationisrecommendedforproductionsetups,itmaybeoverkillinadevelopmentenvironment.Toturnoffauthentication,theoption--server.authenticationcanbesettofalseinArangoDB'sconfigurationfileoronthecommand-line.

WebAdminInterface

TheJavaScriptshellhasbeenremovedfromArangoDB'swebinterface.ThefunctionalitytheshellprovidedisstillfullyavailableintheArangoShell(arangosh)binaryshippedwithArangoDB.

ArangoShellandclienttools


572

TheArangoShell(arangosh)andtheotherclienttoolsbundledwithArangoDBcanonlyconnecttoanArangoDBserverofversion3.0orhigher.TheywillnotconnecttoanArangoDB2.8.ThisisbecausetheserverHTTPAPIshavechangedbetween2.8and3.0,andallclienttoolsusestheseAPIs.

InordertoconnecttoearlierversionsofArangoDBwiththeclienttools,anolderversionoftheclienttoolsneedstobekeptinstalled.

ThepreferrednameforthetemplatestringgeneratorfunctionaqlQueryisnowaqlandisautomaticallyavailableinarangosh.Elsewhere,itcanbeloadedlikeconstaql=require('@arangodb').aql.

Command-lineoptionsadded

Allclienttoolsin3.0provideanoption--server.max-packet-sizeforcontrollingthemaximumsizeofHTTPpacketstobehandledbytheclienttools.Thedefaultvalueis128MB,asinpreviousversionsofArangoDB.Incontrasttopreviousversionsinwhichthevaluewashard-coded,theoptionisnowconfigurable.ItcanbeincreasedtomaketheclienttoolshandleverylargeHTTPresultmessagessentbytheserver.

Command-lineoptionschanged

Forallclienttools,theoption--server.disable-authenticationwasrenamedto--server.authentication.Notethatthemeaningoftheoption--server.authenticationistheoppositeoftheprevious--server.disable-authentication.

Theoption--server.ssl-protocolwasrenamedto--ssl.protocol.Themeaningoftheoptionisunchanged.

Thecommand-lineoption--quietwasremovedfromallclienttoolsexceptarangoshbecauseithadnoeffectinthem.

Arangobench

Inordertomakeitspurposemoreapparenttheformerarangobclienttoolhasbeenrenamedtoarangobenchin3.0.

MiscellaneouschangesThechecksumcalculationalgorithmforthecollection.checksum()methodanditscorrespondingRESTAPIGET/_api/collection/<collection</checksumhaschangedin3.0.Checksumscalculatedin3.0willdifferfromchecksumscalculatedwith2.8orbefore.

TheArangoDBserverin3.0doesnotreadafileENDPOINTScontainingalistofadditionalendpointsonstartup.In2.8thisfilewasautomaticallyreadifpresentinthedatabasedirectory.

Thenamesofthesub-threadsstartedbyArangoDBhavechangedin3.0.ThisisrelevantonLinuxonly,wherethreadscanbenamedandthreadnamesmaybevisibletosystemtoolssuchastopormonitoringsolutions.


573

FeaturesandImprovementsThefollowinglistshowsindetailwhichfeatureshavebeenaddedorimprovedinArangoDB2.8.ArangoDB2.8alsocontainsseveralbugfixesthatarenotlistedhere.Foralistofbugfixes,pleaseconsulttheCHANGELOG.

AQLimprovements

AQLGraphTraversals/PatternMatching

AQLoffersanewfeaturetotraverseoveragraphwithoutwritingJavaScriptfunctionsbutwithalltheotherfeaturesyouknowfromAQL.Forthispurpose,aspecialversionofFORvariableNameINexpressionhasbeenintroduced.

Thisspecialversionhasthefollowingformat:FORvertex-variable,edge-variable,path-variableINtraversal-expression,wheretraversal-expressionhasthefollowingformat:[depth]directionstart-vertexgraph-definitionwiththefollowinginputparameters:

depth(optional):defineshowmanystepsareexecuted.Thevaluecaneitherbeanintegervalue(e.g.3)orarangeofintegervalues(e.g.1..5).Thedefaultis1.direction:defineswhichedgedirectionsarefollowed.CanbeeitherOUTBOUND,INBOUNDorANY.start-vertex:defineswherethetraversalisstarted.Mustbean_idvalueoradocument.graph-definition:defineswhichedgecollectionsareusedforthetraversal.MustbeeitherGRAPHgraph-nameforgraphscreatedwiththegraph-module,oralistofedgecollectionsedge-col1,edge-col2,..edge-colN.

Thethreeoutputvariableshavethefollowingsemantics:

vertex-variable:Thelastvisitedvertex.edge-variable:Thelastvisitededge(optional).path-variable:Thecompletepathfromstart-vertextovertex-variable(optional).

ThetraversalstatementcanbeusedinthesamewayastheoriginalFORvariableNameINexpression,andcanbecombinedwithfiltersandotherAQLconstructs.

AsanexampleonecannowfindthefriendsofafriendforacertainuserwiththisAQLstatement:

FORfoaf,e,pathIN2ANY@startUserGRAPH"relations"

FILTERpath.edges[0].type=="friend"

FILTERpath.edges[1].type=="friend"

FILTERfoaf._id!=@startUser

RETURNDISTINCTfoaf

Optimizerruleshavebeenimplementedtogainperformanceofthetraversalstatement.Theserulesmovefilterstatementsintothetraversalstatements.t.pathswhichcanneverpassthefilterarenotemittedtothevariables.

Asanexampletakethequeryaboveandassumethereareedgesthatdonothavetype=="friend".Ifinthefirstedgestepthereissuchanon-friendedgethesecondstepswillneverbecomputedfortheseedgesastheycannotfulfillthefiltercondition.

ArrayIndexes

Hashindexesandskiplistindexescannowoptionallybedefinedforarrayvaluessothattheyindexindividualarraymembersinsteadoftheentirearrayvalue.

Todefineanindexforarrayvalues,theattributenameisextendedwiththeexpansionoperator[*]intheindexdefinition.

Example:

db._create("posts");

db.posts.ensureHashIndex("tags[*]");

Whengiventhefollowingdocument

WhatsNewin2.8

574

https://github.com/arangodb/arangodb/blob/devel/CHANGELOG

{

"tags":[

"AQL",

"ArangoDB",

"Index"

]

}

thisindexwillnowcontaintheindividualvalues"AQL","ArangoDB"and"Index".

Nowtheindexcanbeusedforfindingalldocumentshaving"ArangoDB"somewhereintheirtagsarrayusingthefollowingAQLquery:

FORdocINposts

FILTER"ArangoDB"INdoc.tags[*]

RETURNdoc

Itisalsopossibletocreateanindexonsub-attributesofarrayvalues.Thismakessensewhentheindexattributeisanarrayofobjects,e.g.

db._drop("posts");

db._create("posts");

db.posts.ensureIndex({type:"hash",fields:["tags[*].name"]});

db.posts.insert({tags:[{name:"AQL"},{name:"ArangoDB"},{name:"Index"}]});

db.posts.insert({tags:[{name:"AQL"},{name:"2.8"}]});

Thefollowingquerywillthenusethearrayindex:

FORdocINposts

FILTER'AQL'INdoc.tags[*].name

RETURNdoc

Arrayvalueswillautomaticallybede-duplicatedbeforebeinginsertedintoanarrayindex.

PleasenotethatfilteringusingarrayindexesonlyworksfromwithinAQLqueriesandonlyifthequeryfiltersontheindexedattributeusingtheINoperator.Theothercomparisonoperators(==,!=,>,>=,<,<=)currentlydonotusearrayindexes.


TheAQLqueryoptimizercannowuseindexesifmultiplefilterconditionsonattributesofthesamecollectionarecombinedwithlogicalORs,andiftheusageofindexeswouldcompletelycovertheseconditions.

Forexample,thefollowingqueriescannowusetwoindependentindexesonvalue1andvalue2(thelatterqueryrequiresthattheindexesareskiplistindexesduetousageofthe<and>comparisonoperators):


FORdocINcollectionFILTERdoc.value1<42||doc.value2>23RETURNdoc

Thenewoptimizerrule"sort-in-values"cannowpre-sorttheright-handsideoperandofINandNOTINoperatorssotheoperationcanuseabinarysearchwithlogarithmiccomplexityinsteadofalinearsearch.Therulewillbeappliedwhentheright-handsideoperandofanINorNOTINoperatorinafilterconditionisavariablethatisdefinedinadifferentloop/scopethantheoperatoritself.Additionally,thefilterconditionmustconsistofsolelytheINorNOTINoperationinordertoavoidanyside-effects.

Therulewillkickinforaqueriessuchasthefollowing:

LETvalues=/*someruntimeexpressionhere*/

FORdocINcollection

FILTERdoc.valueINvalues

RETURNdoc

Itwillnotbeappliedforthefollowigqueries,becausetheright-handsideoperandoftheINiseithernotavariable,orbecausetheFILTERconditionmayhavesideeffects:

WhatsNewin2.8

575

FORdocINcollection

FILTERdoc.valueIN/*someruntimeexpressionhere*/

RETURNdoc

LETvalues=/*someruntimeexpressionhere*/

FORdocINcollection

FILTERFUNCTION(doc.values)==23&&doc.valueINvalues

RETURNdoc

AQLfunctionsadded


POW(base,exponent):returnsthebasetotheexponentexp

UNSET_RECURSIVE(document,attributename,...):recursivelyremovestheattributesattributename(canbeoneormany)fromdocumentanditssub-documents.Allotherattributeswillbepreserved.Multipleattributenamescanbespecifiedbyeitherpassingmultipleindividualstringargumentnames,orbypassinganarrayofattributenames:

UNSET_RECURSIVE(doc,'_id','_key','foo','bar')

UNSET_RECURSIVE(doc,['_id','_key','foo','bar'])

IS_DATESTRING(value):returnstrueifvalueisastringthatcanbeusedinadatefunction.Thisincludespartialdatessuchas2015or2015-10andstringscontaininginvaliddatessuchas2015-02-31.Thefunctionwillreturnfalseforallnon-stringvalues,evenifsomeofthemmaybeusableindatefunctions.


theArangoShellnowprovidestheconveniencefunctiondb._explain(query)forretrievingahuman-readableexplanationofAQLqueries.Thisfunctionisashorthandforrequire("org/arangodb/aql/explainer").explain(query).

theAQLqueryoptimizernowautomaticallyconvertsLENGTH(collection-name)toanoptimizedexpressionthatreturnsthenumberofdocumentsinacollection.PreviousversionsofArangoDBreturnedawarningwhenusingthisexpressionandalsoenumeratedalldocumentsinthecollection,whichwasinefficient.

improvedperformanceofskippingovermanydocumentsinanAQLquerywhennoindexesandnofiltersareused,e.g.

FORdocINcollection

LIMIT1000000,10

RETURNdoc

addedclusterexecutionsiteinfoinexecutionplanexplainoutputforAQLqueries

for30+AQLfunctionsthereisnowanadditionalimplementationinC++thatremovestheneedforinternaldataconversionwhenthefunctioniscalled

theAQLeditorinthewebinterfacenowsupportsusingbindparameters

Deadlockdetection

ArangoDB2.8nowhasanautomaticdeadlockdetectionfortransactions.

Adeadlockisasituationinwhichtwoormoreconcurrentoperations(usertransactionsorAQLqueries)trytoaccessthesameresources(collections,documents)andneedtowaitfortheotherstofinish,butnoneofthemcanmakeanyprogress.

Incaseofsuchadeadlock,therewouldbenoprogressforanyoftheinvolvedtransactions,andnoneoftheinvolvedtransactionscouldevercomplete.Thisiscompletelyundesirable,sothenewautomaticdeadlockdetectionmechanisminArangoDBwillautomaticallykickinandabortoneofthetransactionsinvolvedinsuchadeadlock.Abortingmeansthatallchangesdonebythetransactionwillberolledbackanderror29(deadlockdetected)willbethrown.

WhatsNewin2.8

576

Clientcode(AQLqueries,usertransactions)thataccessesmorethanonecollectionshouldbeawareofthepotentialofdeadlocksandshouldhandletheerror29(deadlockdetected)properly,eitherbypassingtheexceptiontothecallerorretryingtheoperation.

Replication

Thefollowingimprovementsforreplicationhavebeenmadein2.8(note:mostofthemhavebeenbackportedtoArangoDB2.7aswell):

addedautoResyncconfigurationparameterforcontinuousreplication.

Whensettotrue,areplicationslavewillautomaticallytriggerafulldatare-synchronizationwiththemasterwhenthemastercannotprovidethelogdatatheslavehadaskedfor.NotethatautoResyncwillonlyworkwhentheoptionrequireFromPresentisalsosettotrueforthecontinuousreplication,orwhenthecontinuoussyncerisstartedanddetectsthatnostarttickispresent.

Automaticre-synchronizationmaytransferalotofdatafromthemastertotheslaveandmaybeexpensive.Itisthereforeturnedoffbydefault.Whenturnedoff,theslavewillneverperformanautomaticre-synchronizationwiththemaster.

addedidleMinWaitTimeandidleMaxWaitTimeconfigurationparametersforcontinuousreplication.

Theseparameterscanbeusedtocontroltheminimumandmaximumwaittimetheslavewill(intentionally)idleandnotpollformasterlogchangesincasethemasterhadsentthefulllogsalready.TheidleMaxWaitTimevaluewillonlybeusedwhenadapativePollingissettotrue.WhenadaptivePollingisdisabled,onlyidleMinWaitTimewillbeusedasaconstanttimespaninwhichtheslavewillnotpollthemasterforfurtherchanges.Thedefaultvaluesare0.5secondsforidleMinWaitTimeand2.5secondsforidleMaxWaitTime,whichcorrespondtothehard-codedvaluesusedinpreviousversionsofArangoDB.

addedinitialSyncMaxWaitTimeconfigurationparameterforinitialandcontinuousreplication

Thisoptioncontrolsthemaximumwaittime(inseconds)thattheinitialsynchronizationwillwaitforaresponsefromthemasterwhenfetchinginitialcollectiondata.Ifnoresponseisreceivedwithinthistimeperiod,theinitialsynchronizationwillgiveupandfail.ThisoptionisalsorelevantforcontinuousreplicationincaseautoResyncissettotrue,asthenthecontinuousreplicationmaytriggerafulldatare-synchronizationincasethemastercannotthelogdatatheslavehadaskedfor.

HTTPrequestssentfromtheslavetothemasterduringinitialsynchronizationwillnowberetriediftheyfailwithconnectionproblems.

theinitialsynchronizationnowlogsitsprogresssoitcanbequeriedusingtheregularreplicationstatuscheckAPIs.

addedasyncattributeforsyncandsyncCollectionoperationscalledfromtheArangoShell.Setthingthisattributetotruewillmakethesynchronizationjobontheservergointothebackground,sothattheshelldoesnotblock.ThestatusofthestartedasynchronoussynchronizationjobcanbequeriedfromtheArangoShelllikethis:

/*startsinitialsynchronization*/

varreplication=require("org/arangodb/replication");

varid=replication.sync({


username:"myuser",


async:true

});

/*nowquerytheidofthereturnedasyncjobandprintthestatus*/


TheresultofgetSyncResult()willbefalsewhiletheserver-sidejobhasnotcompleted,anddifferenttofalseifithascompleted.Whenithascompleted,alljobresultdetailswillbereturnedbythecalltogetSyncResult().

thewebadmininterfacedashboardnowshowsaserver'sreplicationstatusatthebottomofthepage

WebAdminInterface

Thefollowingimprovementshavebeenmadeforthewebadmininterface:

theAQLeditornowhassupportforbindparameters.Thebindparametervaluescanbeeditedinthewebinterfaceandsavedwithaqueryforfutureuse.

WhatsNewin2.8

577

theAQLeditornowallowscancelingrunningqueries.Thiscanbeusedtocancellong-runningquerieswithoutswitchingtothequerymanagementsection.

thedashboardnowprovidesinformationabouttheserver'sreplicationstatusatthebottomofthepage.Thiscanbeusedtotrackeitherthestatusofaone-timesynchronizationorthecontinuousreplication.

thecompactionstatusandsomestatusinternalsaboutcollectionsarenowdisplayedinthedetailviewforacollectioninthewebinterface.Thesedatacanbeusedfordebuggingcompactionissues.

unloadingacollectionviathewebinterfacewillnowtriggergarbagecollectioninallv8contextsandforceaWALflush.Thisincreasesthechancesofperfomingtheunloadfaster.

thestatusterminologyforcollectionsforwhichanunloadrequesthasbeenissuedviathewebinterfacewaschangedfromintheprocessofbeingunloadedtowillbeunloaded.Thisismoreaccurateastheactualunloadmaybepostponeduntillateriftherearestillreferencespointingtodatainthecollection.

Foxximprovementsthemoduleresolutionusedbyrequirenowbehavesmorelikeinnode.js

theorg/arangodb/requestmodulenowreturnsresponsebodiesforerrorresponsesbydefault.Theoldbehaviorofnotreturningbodiesforerrorresponsescanbere-enabledbyexplicitlysettingtheoptionreturnBodyOnErrortofalse

Miscellaneouschanges

Thestartupoption--server.hide-product-headercanbeusedtomaketheservernotsendtheHTTPresponseheader"Server:ArangoDB"initsHTTPresponses.ThiscanbeusedtoconcealtheservermakefromHTTPclients.Bydefault,theoptionisturnedoffsotheheaderisstillsentasusual.

arangodumpandarangorestorenowhavebettererrorreporting.Additionally,arangodumpwillnowfailbydefaultwhentryingtodumpedgesthatrefertoalreadydroppedcollections.Thiscanbecircumventedbyspecifyingtheoption--forcetruewheninvokingarangodump.

arangoimpnowprovidesanoption--create-collection-typetospecifythetypeofthecollectiontobecreatedwhen--create-collectionissettotrue.Previously--create-collectionalwayscreateddocumentcollectionsandthecreationofedgecollectionswasnotpossible.

WhatsNewin2.8

578


AQL

Keywordsadded

ThefollowingAQLkeywordswereaddedinArangoDB2.8:

GRAPH

OUTBOUND

INBOUND

ANY

ALL

NONE

AGGREGATE

Usageofthesekeywordsforcollectionnames,variablenamesorattributenamesinAQLquerieswillnotbepossiblewithoutquoting.Forexample,thefollowingAQLquerywillstillworkasitusesaquotedcollectionnameandaquotedattributename:

FORdocIN`OUTBOUND`

RETURNdoc.`any`

Changedbehavior

TheAQLfunctionsNEARandWITHINnowhavestrictervalidationsfortheirinputparameterslimit,radiusanddistance.Theymaynowthrowexceptionswheninvalidparametersarepassedthatmayhavenotledtoexceptionsinpreviousversions.

Additionally,theexpansion([*])operatorinAQLhaschangeditsbehaviorwhenhandlingnon-arrayvalues:

InArangoDB2.8,callingtheexpansionoperatoronanon-arrayvaluewillalwaysreturnanemptyarray.PreviousversionsofArangoDBexpandednon-arrayvaluesbycallingtheTO_ARRAY()functionforthevalue,whichforexamplereturnedanarraywithasinglevalueforboolean,numericandstringinputvalues,andanarraywiththeobject'svaluesforanobjectinputvalue.Thisbehaviorwasinconsistentwithhowtheexpansionoperatorworksforthearrayindexesin2.8,sothebehaviorisnowunified:

iftheleft-handsideoperandof[*]isanarray,thearraywillbereturnedasiswhencalling[*]onitiftheleft-handsideoperandof[*]isnotanarray,anemptyarraywillbereturnedby[*]

AQLqueriesthatrelyontheoldbehaviorcanbechangedbyeithercallingTO_ARRAYexplicitlyorbyusingthe[*]atthecorrectposition.

Thefollowingexamplequerywillchangeitsresultin2.8comparedto2.7:

LETvalues="foo"RETURNvalues[*]

In2.7thequeryhasreturnedthearray["foo"],butin2.8itwillreturnanemptyarray[].Tomakeitreturnthearray["foo"]again,anexplicitTO_ARRAYfunctioncallisneededin2.8(whichinthiscaseallowstheremovalofthe[*]operatoraltogether).Thisalsoworksin2.7:

LETvalues="foo"RETURNTO_ARRAY(values)

Anotherexample:

LETvalues=[{name:"foo"},{name:"bar"}]

RETURNvalues[*].name[*]


579

Theabovereturned[["foo"],["bar"]]in2.7.In2.8itwillreturn[[],[]],becausethevalueofname`isnotanarray.Tochangetheresultstothe2.7style,thequerycanbechangedto

LETvalues=[{name:"foo"},{name:"bar"}]

RETURNvalues[*RETURNTO_ARRAY(CURRENT.name)]

Theabovealsoworksin2.7.Thefollowingtypesofquerieswon'tchange:

LETvalues=[1,2,3]RETURNvalues[*]

LETvalues=[{name:"foo"},{name:"bar"}]RETURNvalues[*].name

LETvalues=[{names:["foo","bar"]},{names:["baz"]}]RETURNvalues[*].names[*]

LETvalues=[{names:["foo","bar"]},{names:["baz"]}]RETURNvalues[*].names[**]

Deadlockhandling

Clientapplicationsshouldbepreparedtohandleerror29(deadlockdetected)thatArangoDBmaynowthrowwhenitdetectsadeadlockacrossmultipletransactions.Whenaclientapplicationreceiveserror29,itshouldretrytheoperationthatfailed.

TheerrorcanonlyoccurforAQLqueriesorusertransactionsthatinvolvemorethanasinglecollection.

Optimizer

TheAQLexecutionnodetypeIndexRangeNodewasreplacedwithanewmorecapableexecutionnodetypeIndexNode.ThatmeansinexecutionplanexplainoutputtherewillbenomoreIndexRangeNodesbutonlyIndexNode.Thisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.

TheoptimizerrulethatmakesAQLqueriesactuallyuseindexeswasalsorenamedfromuse-index-rangetouse-indexes.Againthisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.

Thequeryoptimizerruleremove-collect-intowasrenamedtoremove-collect-variables.Thisaffectsexplainoutputthatcanberetrievedviarequire("org/arangodb/aql/explainer").explain(query),db._explain(query),andtheHTTPqueryexplainAPI.

HTTPAPI

Whenaserver-sideoperationgotcanceledduetoanexplicitclientcancelrequestviaHTTPDELETE/_api/job,previousversionsofArangoDBreturnedanHTTPstatuscodeof408(requesttimeout)fortheresponseofthecanceledoperation.

TheHTTPreturncode408hascausedproblemswithsomeclientapplications.Somebrowsers(e.g.Chrome)handleda408responsebyresendingtheoriginalrequest,whichistheoppositeofwhatisdesiredwhenajobshouldbecanceled.

ThereforeArangoDBwillreturnHTTPstatuscode410(gone)forcanceledoperationsfromversion2.8on.

Foxx

ModelandRepository

DuetocompatibilityissuestheModelandRepositorytypesarenolongerimplementedasES2015classes.

Thepre-2.7"extend"stylesubclassingissupportedagainandwillnotemitanydeprecationwarnings.


varMyModel=Foxx.Model.extend({

//...

schema:{/*...*/}

});

Moduleresolution


580

ThebehavioroftheJavaScriptmoduleresolutionusedbytherequirefunctionhasbeenmodifiedtoimprovecompatibilitywithmoduleswrittenforNode.js.

Specifically

absolutepaths(e.g./some/absolute/path)arenowalwaysinterpretedasabsolutefilesystempaths,relativetothefilesystemroot

globalnames(e.g.global/name)arenowfirstintepretedasreferencestomodulesresidinginarelevantnode_modulesfolder,abuilt-inmoduleoramatchingdocumentintheinternal_modulescollection,andonlyresolvedtolocalfilepathsifnoothermatchisfound

Previouslythetwoformatsweretreatedinterchangeablyandwouldberesolvedtolocalfilepathsfirst,leadingtoproblemswhenlocalfilesusedthesamenamesasothermodules(e.g.alocalfilechai.jswouldcauseproblemswhentryingtoloadthechaimoduleinstalledinnode_modules).

Formoreinformationseetheblogannouncementofthischangeandtheupgradeguide.

Moduleorg/arangodb/request

Themodulenowalwaysreturnsresponsebodies,evenforerrorresponses.Inversionspriorto2.8themodulewouldsilentlydropresponsebodiesiftheresponseheaderindicatedanerror.

TheoldbehaviorofnotreturningbodiesforerrorresponsescanberestoredbyexplicitlysettingtheoptionreturnBodyOnErrortofalse:

letresponse=request({

//...

returnBodyOnError:false

});

Garbagecollection

TheV8garbagecollectionstrategywasslightlyadjustedsothatiteventuallyhappensinallV8contextsthatholdV8externalobjects(referencestoArangoDBdocumentsandcollections).Thisenablesabettercleanupoftheseresourcesandpreventsotherprocessessuchascompactionbeingstalledwhilewaitingfortheseresourcestobereleased.

InthiscontextthedefaultvaluefortheJavaScriptgarbagecollectionfrequency(--javascript.gc-frequency)wasalsoincreasedfrom10secondsto15seconds,aslessinternaloperationsinArangoDBarecarriedoutinJavaScript.

Clienttools

arangodumpwillnowfailbydefaultwhentryingtodumpedgesthatrefertoalreadydroppedcollections.Thiscanbecircumventedbyspecifyingtheoption--forcetruewheninvokingarangodump


581

https://www.arangodb.com/2015/11/foxx-module-resolution-will-change-in-2-8/


Performanceimprovements

Indexbuckets

Theprimaryindexesandhashindexesofcollectionscannowbesplitintomultipleindexbuckets.ThisoptionwasavailableforedgeindexesonlyinArangoDB2.6.

Abucketcanbeconsideredacontainerforaspecificrangeofindexvalues.Forprimary,hashandedgeindexes,determiningtheresponsiblebucketforanindexvalueisdonebyhashingtheactualindexvalueandapplyingasimplearithmeticoperationonthehash.

Becauseanindexvaluewillbepresentinatmostonebucketandbucketsareindependent,usingmultiplebucketsprovidesthefollowingbenefits:

initiallybuildingthein-memoryindexdatacanbeparallelizedevenforasingleindex,withonethreadperbucket(orwiththreadsbeingresponsibleformorethanonebucketatatime).Thiscanhelpreducingtheloadingtimeforcollections.

resizinganindexwhenitisabouttorunoutofreservespaceisperformedperbucket.Aseachbucketonlycontainsafractionoftheentireindex,resizingandrehashingabucketismuchfasterandlessintrusivethanresizingandrehashingtheentireindex.

Whencreatingnewcollections,thedefaultnumberofindexbucketsis8sinceArangoDB2.7.Inpreviousversions,thedefaultvaluewas1.Thenumberofbucketscanalsobeadjustedforexistingcollectionssotheycanbenefitfromtheoptimizations.Thenumberofindexbucketscanbesetforacollectionatanytimebyusingacollection'spropertiesfunction:

db.collection.properties({indexBuckets:16});

Thenumberofindexbucketsmustbeapowerof2.

Pleasenotethatforbuildingtheindexdataformultiplebucketsinparallelitisrequiredthatacollectioncontainsasignificantamountofdocumentsbecauseforalownumberofdocumentstheoverheadofparallelizationwilloutweighitsbenefits.Thecurrentthresholdvalueis256kdocuments,butthisvaluemaychangeinfutureversionsofArangoDB.Additionally,theconfigurationoption--database.index-threadswilldeterminehowmanyparallelthreadsmaybeusedforbuildingtheindexdata.

Fasterupdateandremoveoperationsinnon-uniquehashindexes

TheuniquehashindexesinArangoDBprovidedanamortizedO(1)lookup,insert,updateandremoveperformance.Non-uniquehashindexesprovidedamortizedO(1)insertperformance,buthadworseperformanceforupdateandremoveoperationsfornon-uniquevalues.Fordocumentswiththesameindexvalue,theymaintainedalistofcollisions.Whenadocumentwasupdatedorremoved,thatexactdocumenthadtobefoundinthecollisionslistfortheindexvalue.WhilegettingtothestartofthecollisionslistwasO(1),scanningthelisthadO(n)performanceintheworstcase(withnbeingthenumberofdocumentswiththesameindexvalue).Overall,thismadeupdateandremoveoperationsinnon-uniquehashindexesslowiftheindexcontainedmanyduplicatevalues.

ThishasbeenchangedinArangoDB2.7sothatnon-uniquehashindexesnowalsoprovideupdateandremoveoperationswithanamortizedcomplexityofO(1),eveniftherearemanyduplicates.

Resizingnon-uniquehashindexesnowalsodoesn'trequirelookingintothedocumentdata(whichmayinvolveadiskaccess)becausetheindexmaintainssomeinternalcachevalueperdocument.Whenresizingandrehashingtheindex(oranindexbucket),theindexwillfirstcompareonlythecachevaluesbeforepeekingintotheactualdocuments.Thischangecanalsoleadtoreducedindexresizingtimes.

Throughputenhancements

TheArangoDB-internalimplementationsfordispatchingrequests,keepingstatisticsandassigningV8contextstothreadshavebeenimprovedinordertouselesslocks.Thesechangesallowhigherconcurrencyandthroughputinthesecomponents,whichcanalsomaketheserverhandlemorerequestsinagivenperiodoftime.

WhatsNewin2.7

582


Whatgainscanbeexpecteddependsonwhichoperationsareexecuted,buttherearereal-worldcasesinwhichthroughputincreasedbybetween25%and70%whencomparedto2.6.

Madvisehints

TheLinuxvariantforArangoDBprovidestheOSwithmadvisehintsaboutindexmemoryanddatafilememory.Thesehintscanspeedupthingswhenmemoryistight,inparticularatcollectionloadtimebutalsoforrandomaccesseslater.ThereisnoformalguaranteethattheOSactuallyusesthemadvisehintsprovidedbyArangoDB,butactualmeasurementshaveshownimprovementsforloadingbiggercollections.

AQLimprovements

Additionaldatefunctions

ArangoDB2.7providesseveralextraAQLfunctionsfordateandtimecalculationandmanipulation.ThesefunctionswerecontributedbyGitHubusers@[email protected]!

Thefollowingextradatefunctionsareavailablefrom2.7on:

DATE_DAYOFYEAR(date):Returnsthedayofyearnumberofdate.Thereturnvaluesrangefrom1to365,or366inaleapyearrespectively.

DATE_ISOWEEK(date):ReturnstheISOweekdateofdate.Thereturnvaluesrangefrom1to53.Mondayisconsideredthefirstdayoftheweek.Therearenofractionalweeks,thusthelastdaysinDecembermaybelongtothefirstweekofthenextyear,andthefirstdaysinJanuarymaybepartofthepreviousyear'slastweek.

DATE_LEAPYEAR(date):Returnswhethertheyearofdateisaleapyear.

DATE_QUARTER(date):Returnsthequarterofthegivendate(1-based):

1:January,February,March2:April,May,June3:July,August,September4:October,November,December

DATE_DAYS_IN_MONTH(date):Returnsthenumberofdaysindate'smonth(28..31).

DATE_ADD(date,amount,unit):Addsamountgiveninunittodateandreturnsthecalculateddate.

unitcanbeeitherofthefollowingtospecifythetimeunittoaddorsubtract(case-insensitive):

y,year,yearsm,month,monthsw,week,weeksd,day,daysh,hour,hoursi,minute,minutess,second,secondsf,millisecond,milliseconds

amountisthenumberofunitstoadd(positivevalue)orsubtract(negativevalue).

DATE_SUBTRACT(date,amount,unit):Subtractsamountgiveninunitfromdateandreturnsthecalculateddate.

ItworksthesameasDATE_ADD(),exceptthatitsubtracts.ItisequivalenttocallingDATE_ADD()withanegativeamount,exceptthatDATE_SUBTRACT()canalsosubtractISOdurations.NotethatnegativeISOdurationsarenotsupported(i.e.startingwith-P,like-P1Y).

DATE_DIFF(date1,date2,unit,asFloat):Calculatethedifferencebetweentwodatesingiventimeunit,optionallywithdecimalplaces.Returnsanegativevalueifdate1isgreaterthandate2.

DATE_COMPARE(date1,date2,unitRangeStart,unitRangeEnd):Comparetwopartialdatesandreturntrueiftheymatch,falseotherwise.Thepartstocomparearedefinedbyarangeoftimeunits.

WhatsNewin2.7

583

https://www.arangodb.com/2015/08/throughput-enhancements/

Thefullrangeis:years,months,days,hours,minutes,seconds,milliseconds.PasstheunittostartfromasunitRangeStart,andtheunittoendwithasunitRangeEnd.Allunitsinbetweenwillbecompared.LeaveoutunitRangeEndtoonlycompareunitRangeStart.

DATE_FORMAT(date,format):Formatadateaccordingtothegivenformatstring.Itsupportsthefollowingplaceholders(case-insensitive):

%t:timestamp,inmillisecondssincemidnight1970-01-01%z:ISOdate(0000-00-00T00:00:00.000Z)%w:dayofweek(0..6)%y:year(0..9999)%yy:year(00..99),abbreviated(lasttwodigits)%yyyy:year(0000..9999),paddedtolengthof4%yyyyyy:year(-009999..+009999),withsignprefixandpaddedtolengthof6%m:month(1..12)%mm:month(01..12),paddedtolengthof2%d:day(1..31)%dd:day(01..31),paddedtolengthof2%h:hour(0..23)%hh:hour(00..23),paddedtolengthof2%i:minute(0..59)%ii:minute(00..59),paddedtolengthof2%s:second(0..59)%ss:second(00..59),paddedtolengthof2%f:millisecond(0..999)%fff:millisecond(000..999),paddedtolengthof3%x:dayofyear(1..366)%xxx:dayofyear(001..366),paddedtolengthof3%k:ISOweekdate(1..53)%kk:ISOweekdate(01..53),paddedtolengthof2%l:leapyear(0or1)%q:quarter(1..4)%a:daysinmonth(28..31)%mmm:abbreviatedEnglishnameofmonth(Jan..Dec)%mmmm:Englishnameofmonth(January..December)%www:abbreviatedEnglishnameofweekday(Sun..Sat)%wwww:Englishnameofweekday(Sunday..Saturday)%&:specialescapesequenceforrareoccasions%%:literal%%:ignored

RETURNDISTINCT

Toreturnuniquevaluesfromaquery,AQLnowprovidestheDISTINCTkeyword.ItcanbeusedasamodifierforRETURNstatements,asashorteralternativetothealreadyexistingCOLLECTstatement.

Forexample,thefollowingqueryonlyreturnsdistinct(unique)statusattributevaluesfromthecollection:

FORdocINcollection

RETURNDISTINCTdoc.status

RETURNDISTINCTisnotallowedonthetop-levelofaqueryifthereisnoFORloopinfrontofit.RETURNDISTINCTisallowedinsubqueries.

RETURNDISTINCTensuresthatthevaluesreturnedaredistinct(unique),butdoesnotguaranteeanyorderofresults.Inordertohavecertainresultorder,anadditionalSORTstatementmustbeaddedtoaquery.

Shorthandobjectnotation

WhatsNewin2.7

584

AQLnowprovidesashorthandnotationforobjectliteralsinthestyleofES6objectliterals:

LETname="Peter"

LETage=42

RETURN{name,age}

Thisisequivalenttothepreviouslyavailablecanonicalform,whichisstillavailableandsupported:

LETname="Peter"

LETage=42

RETURN{name:name,age:age}

Arrayexpansionimprovements

Thealreadyexisting[*]operatorhasbeenimprovedwithoptionalfilteringandprojectionandlimitcapabilities.

Forexample,considerthefollowingexamplequerythatfiltersvaluesfromanarrayattribute:

FORuINusers

RETURN{

name:u.name,

friends:(

FORfINu.friends

FILTERf.age>u.age

RETURNf.name

)

}

Withthe[*]operator,thisquerycanbesimplifiedto

FORuINusers

RETURN{name:u.name,friends:u.friends[*FILTERCURRENT.age>u.age].name}

Thepseudo-variableCURRENTcanbeusedtoaccessthecurrentarrayelement.TheFILTERconditioncanrefertoCURRENToranyvariablesvalidintheouterscope.

Toreturnaprojectionofthecurrentelement,therecannowbeaninlineRETURN:

FORuINusers

RETURNu.friends[*RETURNCONCAT(CURRENT.name,"isafriendof",u.name)]

whichisthesimplifiedvariantfor:

FORuINusers

RETURN(

FORfriendINu.friends

RETURNCONCAT(friend.name,"isafriendof",u.name)

)

Arraycontraction

Inordertocollapse(orflatten)resultsinnestedarrays,AQLnowprovidesthe[**]operator.Itworkssimilartothe[*]operator,butadditionallycollapsesnestedarrays.Howmanylevelsarecollapsedisdeterminedbytheamountof*charactersused.

Forexample,considerthefollowingquerythatproducesanestedresult:

FORuINusers

RETURNu.friends[*].name

The[**]operatorcannowbeappliedtogetridofthenestedarrayandturnitintoaflatarray.Wesimplyapplythe[**]onthepreviousqueryresult:

WhatsNewin2.7

585

RETURN(

FORuINusersRETURNu.friends[*].name

)[**]

Templatequerystrings

AssemblingquerystringsinJavaScripthasbeenerror-pronewhenusingsimplestringconcatenation,especiallybecauseplainJavaScriptstringsdonothavemultiline-support,andbecauseofpotentialparameterinjectionissues.WhilemultilinequerystringscanbeassembledwithES6templatestringssinceArangoDB2.5,andquerybindparametersaretheresinceArangoDB1.0topreventparameterinjection,therewasnoJavaScript-ysolutiontocombinethese.

ArangoDB2.7nowprovidesanES6templatestringgeneratorfunctionthatcanbeusedtoeasilyandsafelyassembleAQLqueriesfromJavaScript.JavaScriptvariablesandexpressionscanbeusedeasilyusingregularES6templatestringsubstitutions:

letname='test';

letattributeName='_key';

letquery=aqlQuery`FORuINusers

FILTERu.name==${name}

RETURNu.${attributeName}`;

db._query(query);

ThisismorelegiblethanwhenusingaplainJavaScriptstringandalsodoesnotrequiredefiningthebindparametervaluesseparately:

letname='test';

letattributeName='_key';

letquery="FORuINusers"+

"FILTERu.name==@name"+

"RETURNu.@attributeName";

db._query(query,{

name,

attributeName

});

TheaqlQuerytemplatestringgeneratorwillalsohandlecollectionobjectsautomatically:

db._query(aqlQuery`FORuIN${db.users}RETURNu.name`);

NotethatwhiletemplatestringsareavailableintheJavaScriptfunctionsprovidedtobuildqueries,theyaren'tafeatureofAQLitself.AQLcouldalwayshandlemultilinequerystringsandprovidedbindparameters(@...)forseparatingthequerystringandtheparametervalues.TheaqlQuerytemplatestringgeneratorfunctionwilltakecareofthisseparation,too,butwilldoitbehindthescenes.

AQLqueryresultcache

TheAQLqueryresultcachecanoptionallycachethecompleteresultsofallorjustselectedAQLqueries.Itcanbeoperatedinthefollowingmodes:

off:thecacheisdisabled.Noqueryresultswillbestoredon:thecachewillstoretheresultsofallAQLqueriesunlesstheircacheattributeflagissettofalsedemand:thecachewillstoretheresultsofAQLqueriesthathavetheircacheattributesettotrue,butwillignoreallothers

Themodecanbesetatserverstartupusingthe--database.query-cache-modeconfigurationoptionandlaterchangedatruntime.Thedefaultvalueisoff,meaningthatthequeryresultcacheisdisabled.Thisisbecausethecachemayconsumeadditionalmemorytokeepqueryresults,andalsobecauseitmustbeinvalidatedwhenchangeshappenincollectionsforwhichresultshavebeencached.

Thequeryresultcachemaythereforehavepositiveornegativeeffectsonqueryexecutiontimes,dependingontheworkload:itwillnotmakemuchsenseturningonthecacheinwrite-onlyorwrite-mostlyscenarios,butthecachemaybeverybeneficialincaseworkloadsareread-onlyorread-mostly,andqueryarecomplex.

Ifthequerycacheisoperatedindemandmode,itcanbecontrolledperqueryifthecacheshouldbecheckedforaresult.

WhatsNewin2.7

586


Optimizer

TheAQLoptimizerrulepatch-update-statementshasbeenadded.ThisrulecanoptimizecertainAQLUPDATEqueriesthatupdatedocumentsintheacollectionthattheyalsoiterateover.

Forexample,thefollowingqueryreadsdocumentsfromacollectioninordertoupdatethem:

FORdocINcollection

UPDATEdocWITH{newValue:doc.oldValue+1}INcollection

Inthiscase,onlyasinglecollectionisaffectedbythequery,andthereisnoindexlookupinvolvedtofindtheto-be-updateddocuments.Inthiscase,theUPDATEquerydoesnotrequiretakingafull,memory-intensivesnapshotofthecollection,butitcanbeperformedinsmallchunks.Thiscanleadtomemorysavingswhenexecutingsuchqueries.

Functioncallargumentsoptimization

ThisoptimizationwillleadtoargumentsinfunctioncallsinsideAQLqueriesnotbeingcopiedbutbeingpassedbyreference.ThismayspeedupcallstofunctionswithbiggerargumentvaluesorqueriesthatcallAQLfunctionsalotoftimes.

WebAdminInterface

Thewebinterfacenowhasanewdesign.

The"Applications"tabinthewebinterfaceshasbeenrenamedto"Services".

TheArangoDBAPIdocumentationhasbeenmovedfromthe"Tools"menutothe"Links"menu.ThenewdocumentationisbasedonSwagger2.0andopensinaseparatewebpage.

Foxximprovements

ES2015Classes

AllFoxxconstructorshavebeenreplacedwithES2015classesandcanbeextendedusingtheclasssyntax.TheextendmethodisstillsupportedatthemomentbutwillbecomedeprecatedinArangoDB2.8andremovedinArangoDB2.9.

Before:


varMyModel=Foxx.Model.extend({

//...

schema:{/*...*/}

});

After:


classMyModelextendsFoxx.Model{

//...

}

MyModel.prototype.schema={/*...*/};

Confidentialconfiguration

Itisnowpossibletospecifyconfigurationoptionswiththetypepassword.ThepasswordtypeisequivalenttothetexttypebutwillbemaskedinthewebfrontendtopreventaccidentalexposureofconfidentialoptionslikeAPIkeysandpasswordswhenconfiguringyourFoxxapplication.

WhatsNewin2.7

587

Dependencies

Thesyntaxforspecifyingdependenciesinmanifestshasbeenextendedtoallowspecifyingoptionaldependencies.Unmetoptionaldependencieswillnotpreventanappfrombeingmounted.Thetraditionalshorthandsyntaxforspecifyingnon-optionaldependencieswillstillbesupportedintheupcomingversionsofArangoDB.

Before:

{

...

"dependencies":{

"notReallyNeeded":"users:^1.0.0",

"totallyNecessary":"sessions:^1.0.0"

}

}

After:

{

"dependencies":{

"notReallyNeeded":{

"name":"users",

"version":"^1.0.0",

"required":false

},

"totallyNecessary":{

"name":"sessions",

"version":"^1.0.0"

}

}

}

Replication

TheexistingreplicationHTTPAPIhasbeenextendedwithmethodsthatreplicationclientscanusetodeterminewhetheragivendate,identifiedbyatickvalue,isstillpresentonamasterforreplication.BycallingtheseAPIs,clientscanmakeaninformeddecisionaboutwhetherthemastercanstillprovideallmissingdatastartingfromthepointuptowhichtheclienthadalreadysynchronized.Thiscanbehelpfulincaseareplicationclientisre-startedafterapause.

Masterserversnowalsotrackupthepointuptowhichtheyhavesentchangestoclientsforreplication.Thisinformationcanbeusedtodeterminethepointofdatathatreplicationclientshavereceivedfromthemaster,andifandhowfarapproximatelytheylagbehind.

Finally,restartingthereplicationapplieronaslaveserverhasbeenmademorerobustincasetheapplierwasstoppedwhiletherewerependingtransactionsonthemasterserver,andre-startingthereplicationapplierneedstorestorethestateofthesetransactions.

Clienttools

Thefilenamesindumpscreatedbyarangodumpnowcontainnotonlythenameofthedumpedcollection,butalsoanadditional32-digithashvalue.Thisisdonetopreventoverwritingdumpfilesincase-insensitivefilesystemswhenthereexistmultiplecollectionswiththesamename(butwithdifferentcases).

Forexample,ifadatabasehadtwocollectionstestandTest,previousversionsofarangodumpcreatedthefollowingfiles:

test.structure.jsonandtest.data.jsonforcollectiontestTest.structure.jsonandTest.data.jsonforcollectionTest

Thisdidnotworkincase-insensitivefilesystems,becausethefilesforthesecondcollectionwouldhaveoverwrittenthefilesofthefirst.arangodumpin2.7willcreatetheuniquefilesinthiscase,byappendingthe32-digithashvaluetothecollectionnameinallcase.Thesefilenameswillbeunambiguousevenincase-insensitivefilesystems.


WhatsNewin2.7

588

Bettercontrol-Csupportinarangosh

WhenCTRL-Cispressedinarangosh,itwillnowabortthelocallyrunningcommand(ifany).Ifnocommandwasrunning,pressingCTRL-Cwillprinta Cfirst.PressingCTRL-Cagainwillthenquitarangosh.

CTRL-Ccanalsobeusedtoresetthecurrentpromptwhileenteringcomplexnestedobjectswhichspanmultipleinputlines.

CTRL-CsupporthasbeenaddedtotheArangoShellversionsbuiltwithReadline-support(LinuxandMacOSonly).TheWindowsversionofArangoDBusesadifferentlibraryforhandlinginput,andsupportforCTRL-Chasnotbeenaddedthereyet.

Start/stop

LinuxstartupscriptsandsystemdconfigurationforarangodnowtrytoadjusttheNOFILE(numberofopenfiles)limitsfortheprocess.Thelimitvalueissetto131072(128k)whenArangoDBisstartedviastart/stopcommands.

ThiswillpreventarangodrunningoutofavailablefiledescriptorsincaseofmanyparallelHTTPconnectionsorlargecollectionswithmanydatafiles.

Additionally,whenArangoDBisstarted/stoppedmanuallyviathestart/stopcommands,themainprocesswillwaitforupto10secondsafteritforksthesupervisorandarangodchildprocesses.Ifthestartupfailswithinthatperiod,thestart/stopscriptwillfailwithanon-zeroexitcode,allowinganyinvokingscriptstohandlethiserror.Previousversionsalwaysreturnedanexitcodeof0,evenwhenarangodcouldn'tbestarted.

Ifthestartupofthesupervisororarangodisstillongoingafter10seconds,themainprogramwillstillreturnwithexitcode0inordertonotblockanyscripts.Thelimitof10secondsisarbitrarybecausethetimerequiredforanarangodstartupisnotknowninadvance.

Non-sparselogfiles

WALlogfilesanddatafilescreatedbyarangodarenownon-sparse.ThispreventsSIGBUSsignalsbeingraisedwhenamemory-mappedregionbackedbyasparsedatafilewasaccessedandthememoryregionwasnotactuallybackedbydisk,forexamplebecausethediskranoutofspace.

arangodnowalwaysfullyallocatesthediskspacerequiredforalogfileordatafilewhenitcreatesone,sothememoryregioncanalwaysbebackedbydisk,andmemorycanbeaccessedwithoutSIGBUSbeingraised.

WhatsNewin2.7

589


AQLchanges

DISTINCTisnowakeywordinAQL.

AQLqueriesthatuseDISTINCT(inlower,upperormixedcase)asanidentifier(i.e.asavariable,acollectionnameorafunctionname)willstopworking.Tomakesuchqueriesworkingagain,eachoccurrenceofDISTINCTinanAQLqueryshouldbeenclosedinbackticks.ThiswillturnDISTINCTfromakeywordintoanidentifieragain.

TheAQLfunctionSKIPLIST()hasbeenremovedinArangoDB2.7.ThisfunctionwasdeprecatedinArangoDB2.6.Itwasaleft-overfromtimeswhenthequeryoptimizerwasn'tabletouseskiplistindexestogetherwithfilters,skipandlimitvalues.Sincethisissuebeenfixedsinceversion2.3,thereisnoAQLreplacementfunctionforSKIPLIST.QueriesthatusetheSKIPLISTfunctioncanbefixedbyusingtheusualcombinationofFOR,FILTERandLIMIT,e.g.


FILTERdoc.value>=@value

SORTdoc.valueDESC

LIMIT1

RETURNdoc

Foxxchanges

Bundlingandcompilation

TheassetspropertyisnolongersupportedinFoxxmanifestsandisscheduledtoberemovedinafutureversionofArangoDB.Thefilespropertycanstillbeusedtoservestaticassetsbutitisrecommendedtouseseparatetoolingtocompileandbundleyourassets.

Manifestscripts

Thepropertiessetupandteardownhavebeenmovedintothescriptspropertymap:

Before:

{

...


"teardown":"scripts/teardown.js"

}

After:

{

...

"scripts":{


"teardown":"scripts/teardown.js"

}

}

FoxxQueues

Function-basedFoxxQueuejobtypesarenolongersupported.Tolearnabouthowyoucanusethenewscript-basedjobtypesfollowtheupdatedrecipeinthecookbook.


590

https://docs.arangodb.com/2.8/Cookbook/FoxxQueues.html

FoxxSessions

ThejwtandtypeoptionshavebeenremovedfromtheactivateSessionsAPI.

IfyouwanttoreplicatethebehaviorofthejwtoptionyoucanusetheJWTfunctionsinthecryptomodule.AJWT-basedsessionstoragethatdoesn'twritesessionstothedatabaseisavailableasthesessions-jwtappintheFoxxappstore.

Thesessiontypeisnowinferredfromthepresenceofthecookieorheaderoptions(allowingyoutoenablesupportforboth).Ifyouwanttousethedefaultsettingsforcookieorheaderyoucanpassthevaluetrueinstead.

ThesessionStorageAppoptionhasbeenremovedinfavourofthesessionStorageoption.

Before:


varctrl=newFoxx.Controller(applicationContext);


sessionStorageApp:'some-sessions-app',

type:'cookie'

});

After:


sessionStorage:applicationContext.dependencies.sessions.sessionStorage,

cookie:true

});

Requestmodule

Themoduleorg/arangodb/requestusesaninternallibraryfunctionforsendingHTTPrequests.ThislibraryfunctionallyunconditionallysetanHTTPheaderAccept-Encoding:gzipinalloutgoingHTTPrequests,withoutclientcodehavingtosetthisheaderexplicitly.

Thishasbeenfixedin2.7,soAccept-Encoding:gzipisnotsetautomaticallyanymore.AdditionallytheheaderUser-Agent:ArangoDBisnotsetautomaticallyeither.Ifclientapplicationsrelyontheseheadersbeingsent,theyarefreetoadditwhenconstructingrequestsusingtherequestmodule.

Theinternal.download()functionisalsoaffectedbythischange.Again,theheadercanbeaddedhereifrequiredbypassingitviaaheaderssub-attributeinthethirdparameter(options)tothisfunction.

arangodump/backupsThefilenamesindumpscreatedbyarangodumpnowcontainnotonlythenameofthedumpedcollection,butalsoanadditional32-digithashvalue.Thisisdonetopreventoverwritingdumpfilesincase-insensitivefilesystemswhenthereexistmultiplecollectionswiththesamename(butwithdifferentcases).

Thischangeleadstochangedfilenamesindumpscreatedbyarangodump.Ifanyclientscriptsdependonthefilenamesinthedumpoutputdirectorybeingequaltothecollectionnameplusoneofthesuffixes.structure.jsonand.data.json,theyneedtobeadjusted.

StartingwithArangoDB2.7,thefilenameswillcontainanunderscoreplusthe32-digitMD5value(representedinhexadecimalnotation)ofthecollectionname.

Forexample,whenarangodumpdumpsdataoftwocollectionstestandTest,thefilenamesinpreviousversionsofArangoDBwere:

test.structure.json(definitionsforcollectiontest)test.data.json(dataforcollectiontest)Test.structure.json(definitionsforcollectionTest)Test.data.json(dataforcollectionTest)

In2.7,thefilenameswillbe:

test_098f6bcd4621d373cade4e832627b4f6.structure.json(definitionsforcollectiontest)


591

https://github.com/arangodb/foxx-sessions-jwt

test_098f6bcd4621d373cade4e832627b4f6.data.json(dataforcollectiontest)Test_0cbc6611f5540bd0809a388dc95a615b.structure.json(definitionsforcollectionTest)Test_0cbc6611f5540bd0809a388dc95a615b.data.json(dataforcollectionTest)

Starting/stopping

Whenstartingarangod,theserverwillnowdroptheprocessprivilegestothespecifiedvaluesinoptions--server.uidand--server.gidinstantlyafterparsingthestartupoptions.

Thatmeanswheneither--server.uidor--server.gidareset,theprivilegechangewillhappenearlier.Thismaypreventbindingtheservertoanendpointwithaportnumberlowerthan1024ifthearangodbuserhasnoprivilegesforthat.PreviousversionsofArangoDBchangedtheprivilegeslater,sosomestartupactionswerestillcarriedoutundertheinvokinguser(i.e.likelyrootwhenstartedviainit.dorsystemscripts)andespeciallybindingtolowportnumberswasstillpossiblethere.

Thedefaultprivilegesforuserarangodbwillnotbesufficientforbindingtoportnumberslowerthan1024.TohaveanArangoDB2.7bindtoaportnumberlowerthan1024,itneedstobestartedwitheitheradifferentprivilegeduser,ortheprivilegesofthearangodbuserhavetoraisedmanuallybeforehand.

Additionally,LinuxstartupscriptsandsystemdconfigurationforarangodnowwilladjusttheNOFILE(numberofopenfiles)limitsfortheprocess.Thelimitvalueissetto131072(128k)whenArangoDBisstartedviastart/stopcommands.Thegoalofthischangeistopreventarangodfromrunningoutofavailablefiledescriptorsforsocketconnectionsanddatafiles.

Connectionhandling

arangodwillnowactuallycloselingeringclientconnectionswhenidleforatleastthedurationspecifiedinthe--server.keep-alive-timeoutstartupoption.

InpreviousversionsofArangoDB,idleconnectionswerenotclosedbytheserverwhenthetimeoutwasreachedandtheclientwasstillconnected.Nowtheconnectionisproperlyclosedbytheserverincaseoftimeout.Clientapplicationsrelyingontheoldbehaviormaynowneedtoreconnecttotheserverwhentheiridleconnectionstimeoutandgetclosed(note:connectionsbeingidleforalongtimemaybeclosedbytheOSorfirewallsanyway-clientapplicationsshouldbeawareofthatandtrytoreconnect).

Optionchanges

Configureoptionsremoved

Thefollowingoptionsforconfigurehavebeenremovedbecausetheywereunusedorexotic:

--enable-timings

--enable-figures

Startupoptionsadded

Thefollowingconfigurationoptionshavebeenaddedin2.7:

--database.query-cache-max-results:setsthemaximumnumberofresultsinAQLqueryresultcacheperdatabase--database.query-cache-mode:setsthemodefortheAQLqueryresultscache.Possiblevaluesareon,offanddemand.Thedefaultvalueisoff


Simplequeries

Manysimplequeriesprovideaskip()functionthatcanbeusedtoskipoveracertainnumberofdocumentsintheresult.ThisfunctionallowedspecifyingnegativeoffsetsinpreviousversionsofArangoDB.Specifyinganegativeoffsetledtothequeryresultbeingiteratedinreverseorder,soskippingwasperformedfromthebackoftheresult.Asmostsimplequeriesdonotprovideaguaranteedresultorder,


592

skippingfromthebackofaresultwithunspecificorderseemsaratherexoticusecaseandwasremovedtoincreaseconsistencywith

AQL,whichalsodoesnotprovidenegativeskipvalues.

NegativeskipvaluesweredeprecatedinArangoDB2.6.

TasksAPI

TheundocumentedfunctionaddJob()hasbeenremovedfromtheorg/arangodb/tasksmoduleinArangoDB2.7.

RuntimeendpointsmanipulationAPI

ThefollowingHTTPRESTAPImethodsforruntimemanipulationofserverendpointshavebeenremovedinArangoDB2.7:

POST/_api/endpoint:todynamicallyaddanendpointwhiletheserverwasrunningDELETE/_api/endpoint:todynamicallyremoveanendpointwhiletheserverwasrunning

ThischangealsoaffectstheequivalentJavaScriptendpointmanipulationmethodsavailableinFoxx.ThefollowingfunctionshavebeenremovedinArangoDB2.7:

db._configureEndpoint()

db._removeEndpoint()


593


APIsadded

Batchdocumentremovalandlookupcommands

Thefollowingcommandshavebeenaddedforcollectionobjects:

collection.lookupByKeys(keys)collection.removeByKeys(keys)

Thesecommandscanbeusedtoperformmulti-documentlookupandremovaloperationsefficientlyfromtheArangoShell.Theargumenttotheseoperationsisanarrayofdocumentkeys.

ThesecommandscanalsobeusedviatheHTTPRESTAPI.Theirendpointsare:

PUT/_api/simple/lookup-by-keysPUT/_api/simple/remove-by-keys

CollectionexportHTTPRESTAPI

ArangoDBnowprovidesadedicatedcollectionexportAPI,whichcantakesnapshotsofentirecollectionsmoreefficientlythanthegeneral-purposecursorAPI.TheexportAPIisusefultotransferthecontentsofanentirecollectiontoaclientapplication.Itprovidesoptionalfilteringonspecificattributes.

TheexportAPIisavailableatendpointPOST/_api/export?collection=....TheAPIhasthesamereturnvaluestructureasthealreadyestablishedcursorAPI(POST/_api/cursor).

AnintroductiontotheexportAPIisgiveninthisblogpost:http://jsteemann.github.io/blog/2015/04/04/more-efficient-data-exports/

AQLimprovements

EDGESAQLFunction

TheAQLfunctionEDGESgotanewfifthoptionalparameter,whichmustbeanobjectifspecified.Rightnowonlyoneoptionisavailableforit:

includeVerticesthisisabooleanparameterthatallowstomodifytheresultofEDGES().ThedefaultvalueforincludeVerticesisfalse,whichdoesnothaveanyeffect.Settingittotruewillmodifytheresult,suchthatalsotheconnectedverticesarereturnedalongwiththeedges:

{vertex:<vertexDocument>,edge:<edgeDocument>}

SubqueryoptimizationsforAQLqueries

Thisoptimizationavoidscopyingintermediateresultsintosubqueriesthatarenotrequiredbythesubquery.

Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/04/subquery-optimizations/

ReturnvalueoptimizationforAQLqueries

Thisoptimizationavoidscopyingthefinalqueryresultinsidethequery'smainReturnNode.

Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/04/return-value-optimization-for-aql/

WhatsNewin2.6

594


http://jsteemann.github.io/blog/2015/04/04/more-efficient-data-exports/

http://jsteemann.github.io/blog/2015/05/04/subquery-optimizations/

http://jsteemann.github.io/blog/2015/05/04/return-value-optimization-for-aql/

SpeedupAQLqueriescontainingbigINlistsforindexlookups

INlistsusedforindexlookupshadperformanceissuesinpreviousversionsofArangoDB.Theseissueshavebeenaddressedin2.6sousingbiggerINlistsforfilteringismuchfaster.

Abriefdescriptioncanbefoundhere:http://jsteemann.github.io/blog/2015/05/07/in-list-improvements/

AddedalternativeimplementationforAQLCOLLECT

Thealternativemethodusesahashtableforgroupinganddoesnotrequireitsinputelementstobesorted.ItwillbetakenintoaccountbytheoptimizerforCOLLECTstatementsthatdonotuseanINTOclause.

IncaseaCOLLECTstatementcanusethehashtablevariant,theoptimizerwillcreateanextraplanforitatthebeginningoftheplanningphase.Inthisplan,noextraSORTnodewillbeaddedinfrontoftheCOLLECTbecausethehashtablevariantofCOLLECTdoesnotrequiresortedinput.Instead,aSORTnodewillbeaddedafterittosortitsoutput.ThisSORTnodemaybeoptimizedawayagaininlaterstages.Ifthesortorderoftheresultisirrelevanttotheuser,addinganextraSORTnullafterahashCOLLECToperationwillallowtheoptimizertoremovethesortsaltogether.

InadditiontothehashtablevariantofCOLLECT,theoptimizerwillmodifytheoriginalplantousetheregularCOLLECTimplementation.Asthisimplementationrequiressortedinput,theoptimizerwillinsertaSORTnodeinfrontoftheCOLLECT.ThisSORTnodemaybeoptimizedawayinlaterstages.

Thecreatedplanswillthenbeshippedthroughtheregularoptimizationpipeline.Intheend,theoptimizerwillpicktheplanwiththelowestestimatedtotalcostasusual.Thehashtablevariantdoesnotrequireanup-frontsortoftheinput,andwillthusbepreferredovertheregularCOLLECTiftheoptimizerestimatesmanyinputelementsfortheCOLLECTnodeandcannotuseanindextosortthem.

TheoptimizercanbeexplicitlytoldtousetheregularsortedvariantofCOLLECTbysuffixingaCOLLECTstatementwithOPTIONS{"method":"sorted"}.ThiswilloverridetheoptimizerguessworkandonlyproducethesortedvariantofCOLLECT.

AblogpostonthenewCOLLECTimplementationcanbefoundhere:http://jsteemann.github.io/blog/2015/04/22/collecting-with-a-hash-table/

Simplifiedreturnvaluesyntaxfordata-modificationAQLqueries

ArangoDB2.4sinceversionallowstoreturnresultsfromdata-modificationAQLqueries.Thesyntaxforthiswasquitelimitedandverbose:

FORiIN1..10

INSERT{value:i}INtest

LETinserted=NEW

RETURNinserted

TheLETinserted=NEWRETURNinsertedwasrequiredliterallytoreturntheinserteddocuments.Nocalculationscouldbemadeusingtheinserteddocuments.

Thisisnowmoreflexible.Afteradata-modificationclause(e.g.INSERT,UPDATE,REPLACE,REMOVE,UPSERT)therecanfollowanynumberofLETcalculations.Thesecalculationscanrefertothepseudo-valuesOLDandNEWthatarecreatedbythedata-modificationstatements.

Thisallowsreturningprojectionsofinsertedorupdateddocuments,e.g.:

FORiIN1..10

INSERT{value:i}INtest

RETURN{_key:NEW._key,value:i}

Stillnoteveryconstructisallowedafteradata-modificationclause.Forexample,nofunctionscanbecalledthatmayaccessdocuments.

Moreinformationcanbefoundhere:http://jsteemann.github.io/blog/2015/03/27/improvements-for-data-modification-queries/

AddedAQLUPSERTstatement

WhatsNewin2.6

595

http://jsteemann.github.io/blog/2015/05/07/in-list-improvements/

http://jsteemann.github.io/blog/2015/04/22/collecting-with-a-hash-table/

http://jsteemann.github.io/blog/2015/03/27/improvements-for-data-modification-queries/

ThisaddsanUPSERTstatementtoAQLthatisacombinationofbothINSERTandUPDATE/REPLACE.TheUPSERTwillsearchforamatchingdocumentusingauser-providedexample.Ifnodocumentmatchestheexample,theinsertpartoftheUPSERTstatementwillbeexecuted.Ifthereisamatch,theupdate/replacepartwillbecarriedout:

UPSERT{page:'index.html'}/*searchexample*/

INSERT{page:'index.html',pageViews:1}/*insertpart*/

UPDATE{pageViews:OLD.pageViews+1}/*updatepart*/

INpageViews

UPSERTcanbeusedwithanUPDATEorREPLACEclause.TheUPDATEclausewillperformapartialupdateofthefounddocument,whereastheREPLACEclausewillreplacethefounddocumententirely.TheUPDATEorREPLACEpartscanrefertothepseudo-valueOLD,whichcontainsallattributesofthefounddocument.

UPSERTstatementscanoptionallyreturnvalues.Inthefollowingquery,thereturnattributefoundwillreturnthefounddocumentbeforetheUPDATEwasapplied.Ifnodocumentwasfound,foundwillcontainavalueofnull.Theupdatedresultattributewillcontaintheinserted/updateddocument:

UPSERT{page:'index.html'}/*searchexample*/

INSERT{page:'index.html',pageViews:1}/*insertpart*/

UPDATE{pageViews:OLD.pageViews+1}/*updatepart*/

INpageViews

RETURN{found:OLD,updated:NEW}

AmoredetaileddescriptionofUPSERTcanbefoundhere:http://jsteemann.github.io/blog/2015/03/27/preview-of-the-upsert-command/


WhenerrorsoccurinsideAQLuserfunctions,theerrormessagewillnowcontainastacktrace,indicatingthelineofcodeinwhichtheerroroccurred.ThisshouldmakedebuggingAQLuserfunctionseasier.

WebAdminInterfaceArangoDB'sbuilt-inwebinterfacenowusessessions.Sessioninformationisstoredincookies,soclientsusingthewebinterfacemustacceptcookiesinordertouseit.

Thenewstartupoption--server.session-timeoutcanbeusedforadjustingthesessionlifetime.

TheAQLeditorinthewebinterfacenowprovidesanexplainfunctionality,whichcanbeusedforinspectingandperformance-tuningAQLqueries.ThequeryexecutiontimeisnowalsodisplayedintheAQLeditor.

Foxxappsthatrequireconfigurationoraremissingdependenciesarenowindicatedintheappoverviewanddetails.

Foxximprovements

ConfigurationandDependencies

Foxxappmanifestscannowdefineconfigurationoptions,aswellasdependenciesonotherFoxxapps.

AnintroductiontoFoxxconfigurationscanbefoundintheblog:https://www.arangodb.com/2015/05/reusable-foxx-apps-with-configurations/

AndtheblogpostonFoxxdependenciescanbefoundhere:https://www.arangodb.com/2015/05/foxx-dependencies-for-more-composable-foxx-apps/

MochaTests

YoucannowwritetestsforyourFoxxappsusingtheMochatestingframework:https://www.arangodb.com/2015/04/testing-foxx-mocha/

ArecipeforwritingtestsforyourFoxxappscanbefoundinthecookbook:https://docs.arangodb.com/2.8/Cookbook/FoxxTesting.html

WhatsNewin2.6

596

http://jsteemann.github.io/blog/2015/03/27/preview-of-the-upsert-command/

https://www.arangodb.com/2015/05/reusable-foxx-apps-with-configurations/

https://www.arangodb.com/2015/05/foxx-dependencies-for-more-composable-foxx-apps/

https://www.arangodb.com/2015/04/testing-foxx-mocha/

https://docs.arangodb.com/2.8/Cookbook/FoxxTesting.html

APIDocumentation

TheAPIdocumentationhasbeenupdatedtoSwagger2.YoucannowalsomountAPIdocumentationinyourownFoxxapps.

Alsoseetheblogpostintroducingthisfeature:https://www.arangodb.com/2015/05/document-your-foxx-apps-with-swagger-2/

CustomScriptsandFoxxQueue

InadditiontotheexistingsetupandteardownscriptsyoucannowdefinecustomscriptsinyourFoxxmanifestandinvoketheseusingthewebadmininterfaceortheFoxxmanagerCLI.Thesescriptscannowalsotakepositionalargumentsandexportreturnvalues.

JobtypesfortheFoxxQueuecannowbedefinedasascriptnameandappmountpathallowingtheuseofFoxxscriptsasjobtypes.Thepre-2.6jobtypesareknowntocauseissueswhenrestartingtheserverandareerror-prone;westronglyrecommendedconvertinganyexistingjobtypestothenewformat.

Clienttools

Thedefaultconfigurationvaluefortheoption--server.request-timeoutwasincreasedfrom300to1200secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).

Thedefaultconfigurationvaluefortheoption--server.connect-timeoutwasincreasedfrom3to5secondsforclienttools(arangosh,arangoimp,arangodump,arangorestore).

Arangorestore

Theoption--create-databasewasaddedforarangorestore.

Settingthisoptiontotruewillnowcreatethetargetdatabaseifitdoesnotexist.Whencreatingthetargetdatabase,theusernameandpasswordspassedtoarangorestorewillbeusedtocreateaninitialuserforthenewdatabase.

Thedefaultvalueforthisoptionisfalse.

Arangoimp

Arangoimpcannowoptionallyupdateorreplaceexistingdocuments,providedtheimportdatacontainsdocumentswith_keyattributes.

Previously,theimportcouldbeusedforinsertingnewdocumentsonly,andre-insertingadocumentwithanexistingkeywouldhavefailedwithauniquekeyconstraintviolatederror.

Thebehaviorofarangoimp(insert,update,replaceonduplicatekey)cannowbecontrolledwiththeoption--on-duplicate.Theoptioncanhaveoneofthefollowingvalues:

error:whenauniquekeyconstrainterroroccurs,donotimportorupdatethedocumentbutreportanerror.Thisisthedefault.

update:whenauniquekeyconstrainterroroccurs,tryto(partially)updatetheexistingdocumentwiththedataspecifiedintheimport.Thismaystillfailifthedocumentwouldviolatesecondaryuniqueindexes.Onlytheattributespresentintheimportdatawillbeupdatedandotherattributesalreadypresentwillbepreserved.ThenumberofupdateddocumentswillbereportedintheupdatedattributeoftheHTTPAPIresult.

replace:whenauniquekeyconstrainterroroccurs,trytofullyreplacetheexistingdocumentwiththedataspecifiedintheimport.Thismaystillfailifthedocumentwouldviolatesecondaryuniqueindexes.ThenumberofreplaceddocumentswillbereportedintheupdatedattributeoftheHTTPAPIresult.

ignore:whenauniquekeyconstrainterroroccurs,ignorethiserror.Therewillbenoinsert,updateorreplacefortheparticulardocument.IgnoreddocumentswillbereportedseparatelyintheignoredattributeoftheHTTPAPIresult.

Thedefaultvalueiserror.

Afewexamplesforusingarangoimpwiththe--on-duplicateoptioncanbefoundhere:http://jsteemann.github.io/blog/2015/04/14/updating-documents-with-arangoimp/

WhatsNewin2.6

597

https://www.arangodb.com/2015/05/document-your-foxx-apps-with-swagger-2/

http://jsteemann.github.io/blog/2015/04/14/updating-documents-with-arangoimp/


SomeLinux-basedArangoDBpackagesarenowusingtcmallocformemoryallocator.

UpgradedICUlibrarytoversion54.Thisincreasesperformanceinmanyplaces.

Allowtosplitanedgeindexintobucketswhichareresizedindividually.Thedefaultvalueis1,resemblingthepre-2.6behavior.Usingmultiplebucketswillleadtotheindexentriesbeingdistributedtotheindividualbuckets,witheachbucketbeingresponsibleonlyforafractionofthetotalindexentries.Usingmultiplebucketsmayleadtomorefrequentbutmuchfasterindexbucketresizes,andisrecommendedforbiggeredgecollections.

Defaultconfigurationvalueforoption--server.backlog-sizewaschangedfrom10to64.

Defaultconfigurationvalueforoption--database.ignore-datafile-errorswaschangedfromtruetofalse

[email protected]

Fulltextindexcannowindextextvaluescontainedindirectsub-objectsoftheindexedattribute.

PreviousversionsofArangoDBonlyindexedtheattributevalueifitwasastring.Sub-attributesoftheindexattributewereignoredwhenfulltextindexing.

Now,iftheindexattributevalueisanobject,theobject'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.Iftheindexattributevalueisanarray,thearray'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.

Forexample,withafulltextindexpresentonthetranslationsattribute,thefollowingtextvalueswillnowbeindexed:

varc=db._create("example");

c.ensureFulltextIndex("translations");

c.insert({translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}});

c.insert({translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"});

c.insert({translations:["ArangoDB","document","database","Foxx"]});

c.fulltext("translations","лиса").toArray();//returnsonlyfirstdocument

c.fulltext("translations","Fox").toArray();//returnsfirstandseconddocuments

c.fulltext("translations","prefix:Fox").toArray();//returnsallthreedocuments

Addedconfigurationoption--server.foxx-queues-poll-interval

ThisstartupoptioncontrolsthefrequencywithwhichtheFoxxqueuesmanagerischeckingthequeue(orqueues)forjobstobeexecuted.

Thedefaultvalueis1second.Loweringthisvaluewillresultinthequeuemanagerwakingupandcheckingthequeuesmorefrequently,whichmayincreaseCPUusageoftheserver.WhennotusingFoxxqueues,thisvaluecanberaisedtosavesomeCPUtime.

Addedconfigurationoption--server.foxx-queues

ThisstartupoptioncontrolswhethertheFoxxqueuemanagerwillcheckqueueandjobentriesinthe_systemdatabaseonly.RestrictingtheFoxxqueuemanagertothe_systemdatabasewillleadtothequeuemanagerhavingtocheckonlythequeuescollectionofasingledatabase,whereasmakingitcheckthequeuesofalldatabasesmightresultinmoreworktobedoneandmoreCPUtimetobeusedbythequeuemanager.

WhatsNewin2.6

598


Requirements

ArangoDB'sbuilt-inwebinterfacenowusescookiesforsessionmanagement.Sessioninformationidsarestoredincookies,soclientsusingthewebinterfacemustacceptcookiesinordertologinanduseit.

Foxxchanges

FoxxQueues

FoxxQueuejobtypedefinitionswerepreviouslybasedonfunctionsandhadtoberegisteredbeforeuse.Duetochangesin2.5thisresultedinproblemswhenrestartingtheserverordefiningjobtypesincorrectly.

Function-basedjobtypeshavebeendeprecatedin2.6andwillberemovedentirelyin2.7.

Inordertoconvertexistingfunction-basedjobtypestothenewscript-basedjobtypes,createcustomscriptsinyourFoxxappandreferencethembytheirnameandthemountpointoftheapptheyaredefinedin.OfficialjobtypesfromtheFoxxappstorecanbeupgradedbyupgradingfromthe1.xversiontothe2.xversionofthesameapp.

Inordertoupgradequeuedjobstothenewjobtypes,youneedtoupdatethetypepropertyoftheaffectedjobsinthedatabase's_jobssystemcollection.Inordertoseethecollectioninthewebinterfaceyouneedtoenablethecollectiontype"System"inthecollectionlistoptions.

Example:

Before:"type":"mailer.postmark"

After:"type":{"name":"mailer","mount":"/my-postmark-mailer"}

FoxxSessions

Theoptionsjwtandtypeofthecontrollermethodcontroller.activateSessionshavebeendeprecatedin2.6andwillberemovedentirelyin2.7.

IfyouwanttousepureJWTsessions,youcanusethesessions-jwtFoxxappfromtheFoxxappstore.

IfyouwanttouseyourownJWT-basedsessions,youcanusetheJWTfunctionsinthecryptomoduledirectly.

Insteadofusingthetypeoptionyoucanjustusethecookieandheaderoptionsontheirown,whichbothnowacceptthevaluetruetoenablethemwiththeirdefaultconfigurations.

TheoptionsessionStorageApphasbeenrenamedtosessionStorageandnowalsoacceptssessionstoragesdirectly.TheoldoptionsessionStorageAppwillberemovedentirelyin2.7.

Libraries

ThebundledversionofthejoilibraryusedinFoxxwasupgradedtoversion6.0.8.ThismayaffectFoxxapplicationsthatdependonthelibrary.

AQLchanges

AQLLENGTHfunction


599

ThereturnvalueoftheAQLLENGTHfunctionwaschangedifLENGTHisappliedonnullorabooleanvalue:

LENGTH(null)nowreturns0.InpreviousversionsofArangoDB,thisreturned4.

LENGTH(false)nowreturns0.InpreviousversionsofArangoDB,thereturnvaluewas5.

LENGTH(true)nowreturns1.InpreviousversionsofArangoDB,thereturnvaluewas4.

AQLgraphfunctions

In2.6thegraphfunctionsdidundergoaperformancelifting.Duringthisprocesswehadtoadopttheresultformatandtheoptionsforsomeofthem.ManygraphfunctionsnowhaveanoptionincludeDatawhichallowstotriggeriftheresultofthisfunctionshouldcontainfullyextracteddocumentsincludeData:trueoronlythe_idvaluesincludeData:false.Inmostusecasesthe_idissufficienttocontinueandtheextractionofdataisanunnecessaryoperation.TheAQLfunctionssupportingthisadditionaloptionare:

SHORTEST_PATHNEIGHBORSGRAPH_SHORTEST_PATHGRAPH_NEIGHBORSGRAPH_EDGES

FurthermoretheresultSHORTEST_PATHhaschanged.Theoldformatreturnedalistofallverticesonthepath.Optionallyitcouldincludeeachsub-pathforthesevertices.Allofthedocumentswerefullyextracted.Example:

[

{

vertex:{

_id:"vertex/1",

_key:"1",

_rev:"1234"

name:"Alice"

},

path:{

vertices:[

{

_id:"vertex/1",

_key:"1",

_rev:"1234"

name:"Alice"

}

],

edges:[]

}

},

{

vertex:{

_id:"vertex/2",

_key:"2",

_rev:"5678"

name:"Bob"

},

path:{

vertices:[

{

_id:"vertex/1",

_key:"1",

_rev:"1234"

name:"Alice"

},{

_id:"vertex/2",

_key:"2",

_rev:"5678"

name:"Bob"

}

],

edges:[

{

_id:"edge/1",

_key:"1",

_rev:"9876",

type:"loves"


600

}

]

}

}

]

Thenewversionismorecompact.EachSHORTEST_PATHwillonlyreturnonedocumenthavingtheattributesvertices,edges,distance.Thedistanceiscomputedtakingintoaccountthegivenweight.OptionallythedocumentscanbeextractedwithincludeData:trueExample:

{

vertices:[

"vertex/1",

"vertex/2"

],

edges:[

"edge/1"

],

distance:1

}

ThenextfunctionthatreturnsadifferentformatisNEIGHBORS.Since2.5itreturnedanobjectwithedgeandvertexforeachconnectededge.Example:

[

{

vertex:{

_id:"vertex/2",

_key:"2",

_rev:"5678"

name:"Bob"

},

edge:{

_id:"edge/1",

_key:"1",

_rev:"9876",

type:"loves"

}

}

]

With2.6itwillonlyreturnthevertexdirectly,againusingincludeData:true.Bydefaultitwillreturnadistinctsetofneighbors,usingtheoptiondistinct:falsewillincludethesamevertexforeachedgepointingtoit.

Example:

[

"vertex/2"

]

FunctionandAPIchanges

Graphmeasurementsfunctions

AllgraphmeasurementsfunctionsinJavaScriptmodulegeneral-graphthatcalculatedasinglefigurepreviouslyreturnedanarraycontainingjustthefigure.Nowthesefunctionswillreturnthefiguredirectlyandnotputitinsideanarray.

Theaffectedfunctionsare:

graph._absoluteEccentricity

graph._eccentricity

graph._absoluteCloseness

graph._closeness

graph._absoluteBetweenness


601

graph._betweenness

graph._radius

graph._diameter

Clientprogramscallingthesefunctionsshouldbeadjustedsotheyprocessthescalarvaluereturnedbythefunctioninsteadofthepreviousarrayvalue.

CursorAPI

AbatchSizevalue0isnowdisallowedwhencallingthecursorAPIviaHTTPPOST/_api/cursor.

TheHTTPRESTAPIPOST/_api/cursordoesnotacceptabatchSizeparametervalueof0anylonger.Abatchsizeof0nevermademuchsense,butpreviousversionsofArangoDBdidnotcheckforthisvalue.NowcreatingacursorusingabatchSizevalue0willresultinanHTTP400errorresponse.

DocumentURLsreturned

TheRESTAPImethodGET/_api/document?collection=...(thatmethodwillreturnpartialURLstoalldocumentsinthecollection)willnowproperlyprefixdocumentaddressURLswiththecurrentdatabasename.

PreviousversionsofArangoDBreturnedtheURLsstartingwith/_api/butwithoutthecurrentdatabasename,e.g./_api/document/mycollection/mykey.Startingwith2.6,theresponseURLswillincludethedatabasenameaswell,e.g./_db/_system/_api/document/mycollection/mykey.

Fulltextindexing

Fulltextindexeswillnowalsoindextextvaluescontainedindirectsub-objectsoftheindexedattribute.

PreviousversionsofArangoDBonlyindexedtheattributevalueifitwasastring.Sub-attributesoftheindexattributewereignoredwhenfulltextindexing.

Now,iftheindexattributevalueisanobject,theobject'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.Iftheindexattributevalueisanarray,thearray'svalueswilleachbeincludedinthefulltextindexiftheyarestrings.

Deprecatedserverfunctionality

Simplequeries

Thefollowingsimplequeryfunctionsarenowdeprecated:

collection.nearcollection.withincollection.geocollection.fulltextcollection.rangecollection.closedRange

ThisalsoleadtothefollowingRESTAPImethodsbeingdeprecatedfromnowon:

PUT/_api/simple/nearPUT/_api/simple/withinPUT/_api/simple/fulltextPUT/_api/simple/range

ItisrecommendedtoreplacecallstothesefunctionsorAPIswithequivalentAQLqueries,whicharemoreflexiblebecausetheycanbecombinedwithotheroperations:


RETURNdoc


RETURNdoc


602

FORdocINFULLTEXT(@@collection,@attributeName,@queryString,@limit)

RETURNdoc


FILTERdoc.value>=@left&&doc.value<@right

LIMIT@skip,@limit

RETURNdoc`

TheabovesimplequeryfunctionsandRESTAPImethodsmayberemovedinfutureversionsofArangoDB.

UsingnegativevaluesforSimpleQuery.skip()isalsodeprecated.ThisfunctionalitywillberemovedinfutureversionsofArangoDB.

AQLfunctions

TheAQLSKIPLISTfunctionhasbeendeprecatedbecauseitisobsolete.

ThefunctionwasintroducedinolderversionsofArangoDBwithalesspowerfulqueryoptimizertoretrievedatafromaskiplistindexusingaLIMITclause.

Since2.3thesamegoalcanbeachievedbyusingregularAQLconstructs,e.g.


FILTERdoc.value>=@value

SORTdoc.value

LIMIT1

RETURNdoc

Startupoptionchanges

Optionsadded

Thefollowingconfigurationoptionshavebeenaddedin2.6:

--server.session-timeout:allowscontrollingthetimeoutofusersessionsinthewebinterface.Thevalueisspecifiedinseconds.

--server.foxx-queues:controlswhethertheFoxxqueuemanagerwillcheckqueueandjobentries.DisablingthisoptioncanreduceserverloadbutwillpreventjobsaddedtoFoxxqueuesfrombeingprocessedatall.

Thedefaultvalueistrue,enablingtheFoxxqueuesfeature.

--server.foxx-queues-poll-interval:allowsadjustingthefrequencywithwhichtheFoxxqueuesmanagerischeckingthequeue(orqueues)forjobstobeexecuted.

Thedefaultvalueis1second.Loweringthisvaluewillresultinthequeuemanagerwakingupandcheckingthequeuesmorefrequently,whichmayincreaseCPUusageoftheserver.

Note:thisoptiononlyhasaneffectwhen--server.foxx-queuesisnotsettofalse.

Optionsremoved

Thefollowingconfigurationoptionshavebeenremovedin2.6.:

--log.severity:thedocsfor--log.severitymentionedlotsofseverities(e.g.exception,technical,functional,development)butonlyafewseverities(e.g.all,human)wereactuallyused,withhumanbeingthedefaultandallenablingtheadditionalloggingofincomingrequests.

Theoptionpretendedtocontrolalotofthingswhichitactuallydidn't.Additionally,theoption--log.requests-filewasaroundforalongtimealready,alsocontrollingrequestlogging.

Becausethe--log.severityoptioneffectivelydidnotcontrolthatmuch,itwasremoved.Asideeffectofremovingtheoptionisthat2.5installationsstartedwithoption--log.severityallwillnotlogrequestsaftertheupgradeto2.6.Thiscanbeadjustedbysettingthe--log.requests-fileoptioninstead.


603

Defaultvalueschanged

Thedefaultvaluesforthefollowingoptionshavechangedin2.6:

--database.ignore-datafile-errors:thedefaultvalueforthisoptionwaschangedfromtruetofalse.

Ifthenewdefaultvalueoffalseisused,thenarangodwillrefuseloadingcollectionsthatcontaindatafileswithCRCmismatchesorothererrors.Acollectionwithdatafileerrorswillthenbecomeunavailable.Thispreventsfollowuperrorsfromhappening.

Theonlywaytoaccesssuchcollectionistousethedatafiledebugger(arango-dfdb)andtrytorepairortruncatethedatafilewithit.

--server.request-timeout:thedefaultvaluewasincreasedfrom300to1200secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).

--server.connect-timeout:thedefaultvaluewasincreasedfrom3to5secondsforallclienttools(arangosh,arangoimp,arangodump,arangorestore).


604


V8versionupgrade

Thebuilt-inversionofV8hasbeenupgradedfrom3.29.54to3.31.74.1.ThisallowsactivatingadditionalES6(alsodubbedHarmonyorES.next)featuresinArangoDB,bothintheArangoShellandtheArangoDBserver.Theycanbeusedforscriptingandinserver-sideactionssuchasFoxxroutes,traversalsetc.

ThefollowingadditionalES6featuresbecomeavailableinArangoDB2.5bydefault:

iteratorsandgeneratorstemplatestringsenhancedobjectliteralsenhancednumericliteralsblockscopingwithletandconstantvariablesusingconst(note:constantvariablesrequireusingstrictmode,too)additionalstringmethods(suchasstartsWith,repeatetc.)

Indeximprovements

Sparsehashandskiplistindexes

Hashandskiplistindexescanoptionallybemadesparse.Sparseindexesexcludedocumentsinwhichatleastoneoftheindexattributesiseithernotsetorhasavalueofnull.

Assuchdocumentsareexcludedfromsparseindexes,theymaycontainfewerdocumentsthantheirnon-sparsecounterparts.Thisenablesfasterindexingandcanleadtoreducedmemoryusageincasetheindexedattributedoesoccuronlyinsome,butnotalldocumentsofthecollection.Sparseindexeswillalsoreducethenumberofcollisionsinnon-uniquehashindexesincasenon-existingoroptionalattributesareindexed.

Inordertocreateasparseindex,anobjectwiththeattributesparsecanbeaddedtotheindexcreationcommands:

db.collection.ensureHashIndex(attributeName,{sparse:true});

db.collection.ensureHashIndex(attributeName1,attributeName2,{sparse:true});

db.collection.ensureUniqueConstraint(attributeName,{sparse:true});

db.collection.ensureUniqueConstraint(attributeName1,attributeName2,{sparse:true});

db.collection.ensureSkiplist(attributeName,{sparse:true});

db.collection.ensureSkiplist(attributeName1,attributeName2,{sparse:true});

db.collection.ensureUniqueSkiplist(attributeName,{sparse:true});

db.collection.ensureUniqueSkiplist(attributeName1,attributeName2,{sparse:true});

Notethatinplaceoftheabovespecializedindexcreationcommands,itisrecommendedtousethemoregeneralindexcreationcommandensureIndex:

db.collection.ensureIndex({type:"hash",sparse:true,unique:true,fields:[attributeName]});

db.collection.ensureIndex({type:"skiplist",sparse:false,unique:false,fields:["a","b"]});

Whennotexplicitlyset,thesparseattributedefaultstofalsefornewhashorskiplistindexes.

Thiscausesachangeinbehaviorwhencreatingauniquehashindexwithoutspecifyingthesparseflag:in2.4,uniquehashindexeswereimplicitlysparse,alwaysexcludingnullvalues.Therewasnooptiontocontrolthisbehavior,andsparsitywasneithersupportedfornon-uniquehashindexesnorskiplistsin2.4.Thisimplicitsparsityofuniquehashindexeswasconsideredaninconsistency,andthereforethebehaviorwascleanedupin2.5.Asof2.5,indexeswillonlybecreatedsparseifsparsityisexplicitlyrequested.Existinguniquehashindexesfrom2.4orbeforewillautomaticallybemigratedsotheyarestillsparseaftertheupgradeto2.5.

WhatsNewin2.5

605


Geoindexesareimplicitlysparse,meaningdocumentswithouttheindexedlocationattributeorcontaininginvalidlocationcoordinatevalueswillbeexcludedfromtheindexautomatically.Thisisalsoachangewhencomparedtopre-2.5behavior,whendocumentswithmissingorinvalidcoordinatevaluesmayhavecausederrorsoninsertionwhenthegeoindex'uniqueflagwassetanditsignoreNullflagwasnot.Thiswasconfusingandhasbeenrectifiedin2.5.ThemethodensureGeoConstraint()nowdoesthesameasensureGeoIndex().Furthermore,theattributesconstraint,unique,ignoreNullandsparseflagsarenowcompletelyignoredwhencreatinggeoindexes.

Thesameistrueforfulltextindexes.Thereisnoneedtospecifynon-uniquenessorsparsityforgeoorfulltextindexes.

Assparseindexesmayexcludesomedocuments,theycannotbeusedforeverytypeofquery.Sparsehashindexescannotbeusedtofinddocumentsforwhichatleastoneoftheindexedattributeshasavalueofnull.Forexample,thefollowingAQLquerycannotuseasparseindex,evenifonewascreatedonattributeattr:

FORdocIncollection

FILTERdoc.attr==null

RETURNdoc

Ifthelookupvalueisnon-constant,asparseindexmayormaynotbeused,dependingontheothertypesofconditionsinthequery.Iftheoptimizercansafelydeterminethatthelookupvaluecannotbenull,asparseindexmaybeused.Whenuncertain,theoptimizerwillnotmakeuseofasparseindexinaqueryinordertoproducecorrectresults.

Forexample,thefollowingqueriescannotuseasparseindexonattrbecausetheoptimizerwillnotknowbeforehandwhetherthecomparisonvaluesfordoc.attrwillincludenull:

FORdocIncollection

FILTERdoc.attr==SOME_FUNCTION(...)

RETURNdoc

FORotherINotherCollection

FORdocIncollection

FILTERdoc.attr==other.attr

RETURNdoc

Sparseskiplistindexescanbeusedforsortingiftheoptimizercansafelydetectthattheindexrangedoesnotincludenullforanyoftheindexattributes.

Selectivityestimates

Indexesoftypeprimary,edgeandhashnowprovideselectivityestimates.ThesewillbeusedbytheAQLqueryoptimizerwhendecidingaboutindexusage.Usingselectivityestimatescanleadtofasterqueryexecutionwhenmoreselectiveindexesareused.

TheselectivityestimatesarealsoreturnedbytheGET/_api/indexRESTAPImethodinasub-attributeselectivityEstimateforeachindexthatsupportsit.Thisattributewillbeomittedforindexesthatdonotprovideselectivityestimates.Ifprovided,theselectivityestimatewillbeanumericvaluebetween0and1.

Selectivityestimateswillalsobereportedintheresultofcollection.getIndexes()forallindexesthatsupportthis.Ifnoselectivityestimatecanbedeterminedforanindex,theattributeselectivityEstimatewillbeomittedhere,too.

Thewebinterfacealsoshowsselectivityestimatesforeachindexthatsupportsthis.

Currentlythefollowingindextypescanprovideselectivityestimates:

primaryindexedgeindexhashindex(uniqueandnon-unique)

Noselectivityestimateswillbeprovidedforindexeswhenrunninginclustermode.

AQLOptimizerimprovements

Sortremoval

WhatsNewin2.5

606

TheAQLoptimizerrule"use-index-for-sort"willnowremovesortsalsoincaseanon-sortedindex(e.g.ahashindex)isusedforonlyequalitylookupsandallsortattributesarecoveredbytheequalitylookupconditions.

Forexample,inthefollowingquerytheextrasortondoc.valuewillbeoptimizedawayprovidedthereisanindexondoc.value):

FORdocINcollection

FILTERdoc.value==1

SORTdoc.value

RETURNdoc

TheAQLoptimizerrule"use-index-for-sort"nowalsoremovessortincasethesortcriteriaexcludestheleft-mostindexattributes,buttheleft-mostindexattributesareusedbytheindexforequality-onlylookups.

Forexample,inthefollowingquerywithaskiplistindexonvalue1,value2,thesortcanbeoptimizedaway:

FORdocINcollection

FILTERdoc.value1==1

SORTdoc.value2

RETURNdoc

Constantattributepropagation

ThenewAQLoptimizerrulepropagate-constant-attributeswilllookforattributesthatareequality-comparedtoaconstantvalue,andwillpropagatethecomparisonvalueintootherequalitylookups.ThisrulewillonlylookinsideFILTERconditions,andinsertconstantvaluesfoundinFILTERs,too.

Forexample,therulewillinsert42insteadofi.valueinthesecondFILTERofthefollowingquery:

FORiINc1

FORjINc2

FILTERi.value==42

FILTERj.value==i.value

RETURN1

Interleavedprocessing

TheoptimizerwillnowinspectAQLdata-modificationqueriesanddetectifthequery'sdata-modificationpartcanruninlockstepwiththedataretrievalpartofthequery,orifthedataretrievalpartmustbeexecutedandcompletedfirstbeforethedata-modificationcanstart.

Executingbothdataretrievalanddata-modificationinlockstepallowsusingmuchsmallerbuffersforintermediateresults,reducingthememoryusageofqueries.Notallqueriesareeligibleforthisoptimization,andtheoptimizerwillonlyapplytheoptimizationwhenitcansafelydetectthatthedata-modificationpartofthequerywillnotmodifydatatobefoundbytheretrievalpart.

Queryexecutionstatistics

ThefilteredattributewasaddedtoAQLqueryexecutionstatistics.ThevalueofthisattributeindicateshowmanydocumentswerefilteredbyFilterNodesintheAQLquery.NotethatIndexRangeNodescanalsofilterdocumentsbyselectingonlytherequiredrangesfromtheindex.ThefilteredvaluewillnotincludetheworkdonebyIndexRangeNodes,butonlytheworkperformedbyFilterNodes.

Languageimprovements

DynamicattributenamesinAQLobjectliterals

ThischangeallowsusingarbitraryexpressionstoconstructattributenamesinobjectliteralsspecifiedinAQLqueries.Todisambiguateexpressionsandotherunquotedattributenames,dynamicattributenamesneedtobeenclosedinbrackets([and]).

Example:

WhatsNewin2.5

607

FORiIN1..100

RETURN{[CONCAT('value-of-',i)]:i}

AQLfunctions

ThefollowingAQLfunctionswereaddedin2.5:

MD5(value):generatesanMD5hashofvalueSHA1(value):generatesanSHA1hashofvalueRANDOM_TOKEN(length):generatesarandomstringvalueofthespecifiedlength

SimplifyFoxxusage

ThankstoouruserfeedbackwelearnedthatFoxxisapowerful,yetrathercomplicatedconcept.With2.5wemadeitlesscomplicatedwhilekeepingallitsstrength.Thatincludesarewriteofthedocumentationaswellassomecodechangesasfollows:

MovedFoxxapplicationstoadifferentfolder.

Until2.4foxxappswerestoredinthefollowingfolderstructure:<app-path>/databases/<dbname>/<appname>:<appversion>.Thiscausedsometroubleasappswherecachedbasedonnameandversionandupdatesdidnotapply.Alsothepathonfilesystemandtheapp'saccessURLhadnorelationtooneanother.NowthepathonfilesystemisidenticaltotheURL(excepttheappendedAPP):<app-path>/_db/<dbname>/<mointpoint>/APP

RewriteofFoxxrouting

TheroutingofFoxxhasbeenexposedtomajorinternalchangesweadjustedbecauseofuserfeedback.Thisallowsustosetthedevelopmentmodepermountpointwithouthavingtochangepathsandholdappsatseparatelocations.

FoxxDevelopmentmode

Thedevelopmentmodeuseduntil2.4isgone.Ithasbeenreplacedbyamuchmorematureversion.Thisincludesthedeprecationofthejavascript.dev-app-pathparameter,whichisuselesssince2.5.Insteadofhavingtwoseparateappdirectoriesforproductionanddevelopment,appsnowresideinoneplace,whichisusedforproductionaswellasfordevelopment.Appscanstillbeputintodevelopmentmode,changingtheirbehaviorcomparedtoproductionmode.Developmentmodeappsarestillrereadfromdiskateveryrequest,andstilltheyshipmoredebugoutput.

Thischangehasalsomadethestartupoptions--javascript.frontend-development-modeand--javascript.dev-app-pathobsolete.Theformeroptionwillnothaveanyeffectwhenset,andthelatteroptionisonlyreadandusedduringtheupgradeto2.5anddoesnothaveanyeffectslater.

Foxxinstallprocess

InstallingFoxxappshasbeenatwostepprocess:importthemintoArangoDBandmountthemataspecificmountpoint.Theseoperationshavebeenjoinedtogether.Youcaninstallanappatonemountpoint,that'sit.Nofetch,mount,unmount,purgecycleanymore.Thecommandshavebeensimplifiedtojust:

install:getyourFoxxappupandrunninguninstall:shutitdownanderaseitfromdisk

Foxxerroroutput

Until2.4theerrorsproducedbyFoxxwerenotoptimal.Often,theerrormessagewasjustunabletoparsemanifestandcontainedonlyaninternalstacktrace.In2.5wemademajorimprovementsthere,includingamuchmorefinegrainederroroutputthathelpsyoudebugyourFoxxapps.Theerrormessageprintedisnowmuchclosertoitssourceandshouldhelpyoutrackitdown.

AlsoweaddedthedefaulthandlersforunhandlederrorsinFoxxapps:

YouwillgetaniceinternalerrorpagewheneveryourFoxxappiscalledbutwasnotinstalledduetoanyerror

WhatsNewin2.5

608

Youwillgetapropererrormessagewhenhavinganuncaughterrorappearsinanyapproute

InproductionmodethemessagesabovewillNOTcontainanyinformationaboutyourFoxxinternalsandaresafetobeexposedtothirdpartyusers.Indevelopmentmodethemessagesabovewillcontainthestacktrace(ifavailable),makingiteasierforyourin-housedevstotrackdownerrorsintheapplication.

Foxxconsole

WeaddedaconsoleobjecttoFoxxapps.AllFoxxappsnowhaveaconsoleobjectimplementingthefamiliarConsoleAPIintheirglobalscope,whichcanbeusedtologdiagnosticmessagestothedatabase.Thisconsolealsoallowstoreadtheerroroutputofonespecificfoxx.

Foxxrequests

Weaddedorg/arangodb/requestmodule,whichprovidesasimpleAPIformakingHTTPrequeststoexternalservices.ThisisenablesFoxxtobedirectlypartofamicroservicearchitecture.

WhatsNewin2.5

609


Changedbehavior

V8

TheV8versionshippedwithArangoDBwasupgradedfrom3.29.59to3.31.74.1.ThisleadstoadditionalECMAScript6(ES6or"harmony")featuresbeingenabledbydefaultinArangoDB'sscriptingenvironment.

Apartfromthat,achangeintheinterpretationofcommand-lineoptionsbyV8mayaffectusers.ArangoDBpassesthevalueofthecommand-lineoption--javascript.v8-optionstoV8andleavesinterpretationofthecontentstoV8.Forexample,theArangoDBoption--javascript.v8-options="--harmony"couldbeusedtotellV8toenableitsharmonyfeatures.

InArangoDB2.4,thefollowingharmonyoptionsweremadeavailablebyV8:

--harmony_scoping(enableharmonyblockscoping)--harmony_modules(enableharmonymodules(impliesblockscoping))--harmony_proxies(enableharmonyproxies)--harmony_generators(enableharmonygenerators)--harmony_numeric_literals(enableharmonynumericliterals(0o77,0b11))--harmony_strings(enableharmonystring)--harmony_arrays(enableharmonyarrays)--harmony_arrow_functions(enableharmonyarrowfunctions)--harmony_classes(enableharmonyclasses)--harmony_object_literals(enableharmonyobjectliteralextensions)--harmony(enableallharmonyfeatures(exceptproxies))

Therewastheoption--harmony,whichturnedonalmostallharmonyfeatures.

InArangoDB2.5,V8providesthefollowingharmony-relatedoptions:

--harmony(enableallcompletedharmonyfeatures)--harmony_shipping(enableallshippedharmonyfeatures)--harmony_modules(enable"harmonymodules(impliesblockscoping)"(inprogress))--harmony_arrays(enable"harmonyarraymethods"(inprogress))--harmony_array_includes(enable"harmonyArray.prototype.includes"(inprogress))--harmony_regexps(enable"harmonyregularexpressionextensions"(inprogress))--harmony_arrow_functions(enable"harmonyarrowfunctions"(inprogress))--harmony_proxies(enable"harmonyproxies"(inprogress))--harmony_sloppy(enable"harmonyfeaturesinsloppymode"(inprogress))--harmony_unicode(enable"harmonyunicodeescapes"(inprogress))--harmony_tostring(enable"harmonytoString")--harmony_numeric_literals(enable"harmonynumericliterals")--harmony_strings(enable"harmonystringmethods")--harmony_scoping(enable"harmonyblockscoping")--harmony_classes(enable"harmonyclasses(impliesblockscoping&objectliteralextension)")--harmony_object_literals(enable"harmonyobjectliteralextensions")--harmony_templates(enable"harmonytemplateliterals")

Notethatthereareextraoptionsforbettercontrollingthededicatedfeatures,andespeciallythatthemeaningofthe--harmonyoptionhaschangedfromenablingallharmonyfeaturestoallcompletedharmonyfeatures!

Usersshouldadjustthevalueof--javascript.v8-optionsaccordingly.

PleasenotethatincompleteharmonyfeaturesaresubjecttochangeinfutureV8releases.


610

Sparseindexes

Hashindexesandskiplistindexescannowbecreatedinasparsevariant.Whennotexplicitlyset,thesparseattributedefaultstofalsefornewindexes.

Thiscausesachangeinbehaviorwhencreatingauniquehashindexwithoutspecifyingthesparseflag.Theuniquehashindexwillbecreatedinanon-sparsevariantinArangoDB2.5.

In2.4andbefore,uniquehashindexeswereimplicitlysparse,alwaysexcludingnullvaluesfromtheindex.Therewasnooptiontocontrolthisbehavior,andsparsitywasneithersupportedfornon-uniquehashindexesnorskiplistsin2.4.Thisimplicitsparsityofjustuniquehashindexeswasconsideredaninconsistency,andthereforethebehaviorwascleanedupin2.5.

Asof2.5,hashandskiplistindexeswillonlybecreatedsparseifsparsityisexplicitlyrequested.Thismayrequireachangeinindex-creatingclientcode,butonlyiftheclientcodecreatesuniquehashindexesandiftheyarestillintendedtobesparse.Inthiscase,theclientcodeshouldexplicitlysetthesparseflagtotruewhencreatingauniquehashindex.

Existinguniquehashindexesfrom2.4orbeforewillautomaticallybemigratedsotheyarestillsparseaftertheupgradeto2.5.Fortheseindexes,thesparseattributewillbepopulatedautomaticallywithavalueoftrue.

Geoindexesareimplicitlysparse,meaningdocumentswithouttheindexedlocationattributeorcontaininginvalidlocationcoordinatevalueswillbeexcludedfromtheindexautomatically.Thisisalsoachangewhencomparedtopre-2.5behavior,whendocumentswithmissingorinvalidcoordinatevaluesmayhavecausederrorsoninsertionwhenthegeoindex'uniqueflagwassetanditsignoreNullflagwasnot.

Thiswasconfusingandhasbeenrectifiedin2.5.ThemethodensureGeoConstraint()nowdoesthesameasensureGeoIndex().Furthermore,theattributesconstraint,unique,ignoreNullandsparseflagsarenowcompletelyignoredwhencreatinggeoindexes.ClientindexcreationcodethereforedoesnotneedtosettheignoreNullorconstraintattributeswhencreatingageoindex.

Thesameistrueforfulltextindexes.Thereisnoneedtospecifynon-uniquenessorsparsityforgeoorfulltextindexes.Theywillalwaysbenon-uniqueandsparse.

MovedFoxxapplicationstoadifferentfolder.

Until2.4foxxappswerestoredinthefollowingfolderstructure:<app-path>/databases/<dbname>/<appname>:<appversion>.Thiscausedsometroubleasappswherecachedbasedonnameandversionandupdatesdidnotapply.Alsothepathonfilesystemandtheapp'saccessURLhadnorelationtooneanother.NowthepathonfilesystemisidenticaltotheURL(excepttheappendedAPP):<app-path>/_db/<dbname>/<mointpoint>/APP

FoxxDevelopmentmode

Thedevelopmentmodeuseduntil2.4isgone.Ithasbeenreplacedbyamuchmorematureversion.Thisincludesthedeprecationofthejavascript.dev-app-pathparameter,whichisuselesssince2.5.Insteadofhavingtwoseparateappdirectoriesforproductionanddevelopment,appsnowresideinoneplace,whichisusedforproductionaswellasfordevelopment.Appscanstillbeputintodevelopmentmode,changingtheirbehaviorcomparedtoproductionmode.Developmentmodeappsarestillrereadfromdiskateveryrequest,andstilltheyshipmoredebugoutput.

Thischangehasalsomadethestartupoptions--javascript.frontend-development-modeand--javascript.dev-app-pathobsolete.Theformeroptionwillnothaveanyeffectwhenset,andthelatteroptionisonlyreadandusedduringtheupgradeto2.5anddoesnothaveanyeffectslater.

Foxxinstallprocess

InstallingFoxxappshasbeenatwostepprocess:importthemintoArangoDBandmountthemataspecificmountpoint.Theseoperationshavebeenjoinedtogether.Youcaninstallanappatonemountpoint,that'sit.Nofetch,mount,unmount,purgecycleanymore.Thecommandshavebeensimplifiedtojust:

install:getyourFoxxappupandrunninguninstall:shutitdownanderaseitfromdiskDeprecatedfeatures

Foxx:methodModel#toJSONSchema(id)isdeprecated,itwillraiseawarningifyouuseit.PleaseuseFoxx.toJSONSchema(id,model)


611

instead.

Removedfeatures

Startupswitch--javascript.frontend-development-mode:Itsmajorpurposewasinternaldevelopmentanyway.Nowthewebfrontendcanbesettodevelopmentmodesimilartoanyotherfoxxapp.Startupswitch--javascript.dev-app-path:WasusedforthedevelopmentmodeofFoxx.Thisisintegratedwiththenormalapp-pathnowandcanbetriggeredonapplevel.Thesecondapp-pathissuperfluous.Foxx:controller.collection:PleaseuseappContext.collectioninstead.Foxx:FoxxRepository.modelPrototype:PleaseuseFoxxRepository.modelinstead.Foxx:Model.extend({},{attributes:{}}):PleaseuseModel.extend({schema:{}})instead.Foxx:requestContext.bodyParam(paramName,description,Model):PleaseuserequestContext.bodyParam(paramName,options)instead.Foxx:requestContext.queryParam({type:string}):PleaseuserequestContext.queryParam({type:joi})instead.Foxx:requestContext.pathParam({type:string}):PleaseuserequestContext.pathParam({type:joi})instead.Graph:Themodulesorg/arangodb/graphandorg/arangodb/graph-blueprint:Pleaseusemoduleorg/arangodb/general-graphinstead.NOTE:Thisdoesnotmeanwedonotsupportblueprintsanymore.Generalgraphcoverseverythingthegraph--blueprintdid,plusmanymorefeatures.General-Graph:Inthemoduleorg/arangodb/general-graphthefunctions_undirectedRelationand_directedRelationarenolongeravailable.Bothfunctionshavebeenunifiedto_relation.


612


V8versionupgrade

Thebuilt-inversionofV8hasbeenupgradedfrom3.16.14to3.29.59.ThisactivatesseveralES6(alsodubbedHarmonyorES.next)featuresinArangoDB,bothintheArangoShellandtheArangoDBserver.Theycanbeusedforscriptingandinserver-sideactionssuchasFoxxroutes,traversalsetc.

ThefollowingES6featuresareavailableinArangoDB2.4bydefault:

iteratorstheofoperatorsymbolspredefinedcollectionstypes(Map,Setetc.)typedarrays

ManyotherES6featuresaredisabledbydefault,butcanbemadeavailablebystartingarangodorarangoshwiththeappropriateoptions:

arrowfunctionsproxiesgeneratorsString,Array,andNumberenhancementsconstantsenhancedobjectandnumericliterals

ToactivatealltheseES6featuresinarangodorarangosh,startitwiththefollowingoptions:

arangosh--javascript.v8-options="--harmony--harmony_generators"

MoredetailsontheavailableES6featurescanbefoundinthisblog.

FoxxGeneratorArangoDB2.4isshippedwithFoxxGenerator,aframeworkforbuildingstandardizedHypermediaAPIseasily.ThegeneratedAPIscanbeconsumedwithclienttoolsthatunderstandSiren.

HypermediaisthesimpleideathatourHTTPAPIsshouldhavelinksbetweentheirendpointsinthesamewaythatourwebsiteshavelinksbetweenthem.FoxxGeneratorisbasedontheideathatyoucanrepresentanAPIasastatechart:Everyendpointisastateandthelinksarethetransitionsbetweenthem.Usingyourdescriptionofstatesandtransitions,itcanthencreateanAPIforyou.

TheFoxxGeneratorcancreateAPIsbasedonasemanticdescriptionofentitiesandtransitions.AblogseriesontheusecasesandhowtousetheFoxxgeneratorishere:

part1part2part3

AcookbookrecipeforgettingstartedwithFoxxGeneratorishere.

AQLimprovements


WhatsNewin2.4

613


https://jsteemann.github.io/blog/2014/12/19/using-es6-features-in-arangodb/

https://github.com/kevinswiber/siren

https://www.arangodb.com/2014/11/26/building-hypermedia-api-json

https://www.arangodb.com/2014/12/02/building-hypermedia-apis-design

https://www.arangodb.com/2014/12/08/building-hypermedia-apis-foxxgenerator

https://docs.arangodb.com/2.8/Cookbook/FoxxGeneratorFirstSteps.html

TheAQLoptimizerhasbeenenhancedtouseofindexesinqueriesinseveraladditionalcases.FilterscontainingtheINoperatorcannowmakeuseofindexes,andmultipleOR-orAND-combinedfilterconditionscannowalsouseindexesifthefilterconditionsrefertothesameindexedattribute.

Hereareafewexamplesofqueriesthatcannowuseindexesbutcouldn'tbefore:

FORdocINcollection

FILTERdoc.indexedAttribute==1||doc.indexedAttribute>99

RETURNdoc

FORdocINcollection

FILTERdoc.indexedAttributeIN[3,42]||doc.indexedAttribute>99

RETURNdoc

FORdocINcollection

FILTER(doc.indexedAttribute>2&&doc.indexedAttribute<10)||

(doc.indexedAttribute>23&&doc.indexedAttribute<42)

RETURNdoc

Additionally,theoptimizerruleremove-filter-covered-by-indexhasbeenadded.ThisruleremovesFilterNodesandCalculationNodesfromanexecutionplanifthefilterconditionisalreadycoveredbyapreviousIndexRangeNode.Removingthefilter'sCalculationNodeandtheFilterNodeitselfwillspeedupqueryexecutionbecausethequeryrequireslesscomputation.

Furthermore,thenewoptimizerruleremove-sort-randwillremoveaSORTRAND()statementandmovetherandomiterationintotheappropriateEnumerateCollectionNode.Thisisusuallymoreefficientthanindividuallyenumeratingandsorting.

Data-modificationqueriesreturningdocuments

INSERT,REMOVE,UPDATEorREPLACEqueriesnowcanoptionallyreturnthedocumentsinserted,removed,updated,orreplaced.Thisishelpfulfortrackingtheauto-generatedattributes(e.g._key,_rev)createdbyanINSERTandinalotofothersituations.

Inordertoreturndocumentsfromadata-modificationquery,thestatementmustimmediatelybeimmediatelyfollowedbyaLETstatementthatassignseitherthepseudo-valueNEWorOLDtoavariable.ThisLETstatementmustbefollowedbyaRETURNstatementthatreturnsthevariableintroducedbyLET:

FORiIN1..100

INSERT{value:i}INtestLETinserted=NEWRETURNinserted

FORuINusers

FILTERu.status=='deleted'

REMOVEuINusersLETremoved=OLDRETURNremoved

FORuINusers

FILTERu.status=='notactive'

UPDATEuWITH{status:'inactive'}INusersLETupdated=NEWRETURNupdated

NEWreferstotheinsertedormodifieddocumentrevision,andOLDreferstothedocumentrevisionbeforeupdateorremoval.INSERTstatementscanonlyrefertotheNEWpseudo-value,andREMOVEoperationsonlytoOLD.UPDATEandREPLACEcanrefertoeither.

Inallcasesthefulldocumentswillbereturnedwithalltheirattributes,includingthepotentiallyauto-generatedattributessuchas_id,_key,or_revandtheattributesnotspecifiedintheupdateexpressionofapartialupdate.


COUNTclause

AnoptionalCOUNTclausewasaddedtotheCOLLECTstatement.TheCOUNTclauseallowsformoreefficientcountingofvalues.

InpreviousversionsofArangoDBonehadtowritethefollowingtocountdocuments:

RETURNLENGTH(

FORdocINcollection

FILTER...somecondition...

RETURNdoc

)

WhatsNewin2.4

614

WiththeCOUNTclause,thequerycanbemodifiedto

FORdocINcollection


COLLECTWITHCOUNTINTOlength

RETURNlength

ThelatterquerywillbemuchmoreefficientbecauseitwillnotproduceanyintermediateresultswithneedtobeshippedfromasubqueryintotheLENGTHfunction.

TheCOUNTclausecanalsobeusedtocountthenumberofitemsineachgroup:

FORdocINcollection


COLLECTgroup=doc.groupWITHCOUNTINTOlength

return{group:group,length:length}

COLLECTmodifications

InArangoDB2.4,COLLECToperationscanbemademoreefficientifonlyasmallfragmentofthegroupvaluesisneededlater.Forthesecases,COLLECTprovidesanoptionalconversionexpressionfortheINTOclause.Thisexpressioncontrolsthevaluethatisinsertedintothearrayofgroupvalues.Itcanbeusedforprojections.

ThefollowingqueryonlycopiesthedateRegisteredattributeofeachdocumentintothegroups,potentiallysavingalotofmemoryandcomputationtimecomparedtocopyingdoccompletely:

FORdocINcollection


COLLECTgroup=doc.groupINTOdates=doc.dateRegistered

return{group:group,maxDate:MAX(dates)}

Comparethistothefollowingvariantofthequery,whichwastheonlywaytoachievethesameresultinpreviousversionsofArangoDB:

FORdocINcollection


COLLECTgroup=doc.groupINTOdates

return{group:group,maxDate:MAX(dates[*].doc.dateRegistered)}

Theabovequerywillneedtocopythefulldocattributeintothelengthsvariable,whereasthenewvariantwillonlycopythedateRegisteredattributeofeachdoc.

Subquerysyntax

InpreviousversionsofArangoDB,subqueriesrequiredextraparenthesesaroundthem,andthiscausedconfusionwhensubquerieswereusedasfunctionparameters.Forexample,thefollowingquerydidnotwork:

LETvalues=LENGTH(

FORdocINcollectionRETURNdoc

)

buthadtobewrittenasfollows:

LETvalues=LENGTH((

FORdocINcollectionRETURNdoc

))

Thiswasunintuitiveandisfixedinversion2.4sothatbothvariantsofthequeryareacceptedandproducethesameresult.

Webinterface

WhatsNewin2.4

615

TheApplicationstabforFoxxapplicationsinthewebinterfacehasgotacompleteredesign.

ItwillnowonlyshowapplicationsthatarecurrentlyrunninginArangoDB.Foraselectedapplication,anewdetailedviewhasbeencreated.Thisviewprovidesabetteroverviewoftheapp,e.g.:

authorlicenseversioncontributorsdownloadlinksAPIdocumentation

InstallinganewFoxxapplicationontheserverismadeeasyusingthenewAddapplicationbutton.TheAddapplicationdialogprovidesallthefeaturesalreadyavailableinthefoxx-managerconsoleapplicationplussomemore:

installaFoxxapplicationfromGithubinstallaFoxxapplicationfromazipfileinstallaFoxxapplicationfromArangoDB'sapplicationstorecreateanewFoxxapplicationfromscratch:thisfeatureusesageneratortocreateaFoxxapplicationwithpre-definedCRUDmethodsforagivenlistofcollections.ThegeneratedFoxxappcaneitherbedownloadedasazipfileorbeinstalledontheserver.StartingwithanewFoxxapphasneverbeeneasier.


Defaultendpointis127.0.0.1

ThedefaultendpointfortheArangoDBserverhasbeenchangedfrom0.0.0.0to127.0.0.1.ThiswillmakenewArangoDBinstallationsunaccessiblefromclientsotherthanlocalhostunlesstheconfigurationischanged.Thisisasecurityprecautionmeasurethathasbeenrequestedasafeaturealotoftimes.

Ifyouarethedevelopmentoption--enable-relative,theendpointwillstillbe0.0.0.0.

Systemcollectionsinreplication

Bydefault,systemcollectionsarenowincludedinreplicationandallreplicationAPIreturnvalues.Thiswillleadtouseraccountsandcredentialsdatabeingreplicatedfrommastertoslaveservers.Thismayoverwriteslave-specificdatabaseusers.

Ifthisisundesired,the_userscollectioncanbeexcludedfromreplicationeasilybysettingtheincludeSystemattributetofalseinthefollowingcommands:

replication.sync({includeSystem:false});replication.applier.properties({includeSystem:false});

Thiswillexcludeallsystemcollections(including_aqlfunctions,_graphsetc.)fromtheinitialsynchronizationandthecontinuousreplication.

Ifthisisalsoundesired,itisalsopossibletospecifyalistofcollectionstoexcludefromtheinitialsynchronizationandthecontinuousreplicationusingtherestrictCollectionsattribute,e.g.:

require("org/arangodb/replication").applier.properties({

includeSystem:true,

restrictType:"exclude",

restrictCollections:["_users","_graphs","foo"]

});

WhatsNewin2.4

616


Changedbehavior

V8upgrade

ThebundledV8versionhasbeenupgradedfrom3.16.14to3.29.59.

Thenewversionprovidesbettererrorchecking,whichcanleadtosubtlechangesintheexecutionofJavaScriptcode.

Thefollowingcode,thoughnonsense,runswithouterrorin2.3and2.4whenstrictmodeisnotenabled:

(function(){

a=true;

a.foo=1;

})();

Whenenablingstrictmode,thefunctionwillthrowanerrorin2.4butnotin2.3:

(function(){

"usestrict";

a=true;

a.foo=1;

})();

TypeError:Cannotassigntoreadonlyproperty'foo'oftrue

Thoughthisisachangeinbehavioritcanbeconsideredanimprovement.Thenewversionactuallyuncoversanerrorthatwentundetectedintheoldversion.

Errormessageshavealsochangedslightlyinthenewversion.ApplicationsthatrelyontheexacterrormessagesoftheJavaScriptenginemayneedtobeadjustedsotheylookfortheupdatederrormessages.

Defaultendpoint

Thedefaultendpointforarangodisnow127.0.0.1.

ThischangewillmodifytheIPaddressArangoDBlistensonto127.0.0.1bydefault.ThiswillmakenewArangoDBinstallationsunaccessiblefromclientsotherthanlocalhostunlesstheconfigurationischanged.Thisisasecurityfeature.

TomakeArangoDBaccessiblefromanyclient,changetheserver'sconfiguration(--server.endpoint)toeithertcp://0.0.0.0:8529ortheserver'spubliclyvisibleIPaddress.

Replication

SystemcollectionsarenowincludedinthereplicationandallreplicationAPIreturnvaluesbydefault.

Thiswillleadtouseraccountsandcredentialsdatabeingreplicatedfrommastertoslaveservers.Thismayoverwriteslave-specificdatabaseusers.

Thismaybeconsideredafeatureorananti-feature,soitisconfigurable.

Ifreplicationofsystemcollectionsisundesired,theycanbeexcludedfromreplicationbysettingtheincludeSystemattributetofalseinthefollowingcommands:

initialsynchronization:replication.sync({includeSystem:false})continuousreplication:replication.applier.properties({includeSystem:false})


617

Thiswillexcludeallsystemcollections(including_aqlfunctions,_graphsetc.)fromtheinitialsynchronizationandthecontinuousreplication.

Ifthisisalsoundesired,itisalsopossibletospecifyalistofcollectionstoexcludefromtheinitialsynchronizationandthecontinuousreplicationusingtherestrictCollectionsattribute,e.g.:

require("org/arangodb/replication").applier.properties({

includeSystem:true,

restrictType:"exclude",

restrictCollections:["_users","_graphs","foo"]

});

Theaboveexamplewillingeneralincludesystemcollections,butwillexcludethespecifiedthreecollectionsfromcontinuousreplication.

TheHTTPRESTAPImethodsforfetchingthereplicationinventoryandfordumpingcollectionsalsosupporttheincludeSystemcontrolflagviaaURLparameterofthesamename.

Buildprocesschanges

Severaloptionsfortheconfigurecommandhavebeenremovedin2.4.Theoptions

--enable-all-in-one-v8

--enable-all-in-one-icu

--enable-all-in-one-libev

--with-libev=DIR

--with-libev-lib=DIR

--with-v8=DIR

--with-v8-lib=DIR

--with-icu-config=FILE

arenotavailableanymorebecausethebuildprocesswillalwaysusethebundledversionsofthelibraries.

WhenbuildingArangoDBfromsourceinadirectorythatalreadycontainedapre-2.4version,itwillbenecessarytorunamakesupercleancommandonceandafullrebuildafterwards:

gitpull

makesuperclean

makesetup

./configure<optionsgohere>

make

MiscellaneouschangesAsaconsequenceofglobalrenaminginthecodebase,theoptionmergeArrayshasbeenrenamedtomergeObjects.ThisoptioncontrolswhetherJSONobjectswillbemergedonanupdateoperationoroverwritten.Thedefaulthasbeen,andstillis,tomerge.Notspecifyingtheparameterwillleadtoamerge,asithasbeenthebehaviorinArangoDBeversince.

ThisaffectstheHTTPRESTAPImethodPATCH/_api/document/collection/key.ItsoptionalURLparametermergeArraysfortheoptionhasbeenrenamedtomergeObjects.

TheAQLUPDATEstatementisalsoaffected,asitsoptionmergeArrayshasalsobeenrenamedtomergeObjects.The2.3query

UPDATEdocINcollectionWITH{...}INcollectionOPTIONS{mergeArrays:false}

shouldthusberewrittentothefollowingin2.4:

UPDATEdocINcollectionWITH{...}INcollectionOPTIONS{mergeObjects:false}

Deprecatedfeatures


618

ForFoxxControllerobjects,themethodcollection()isdeprecatedandwillberemovedinfutureversionofArangoDB.Usingthismethodwillissueawarning.PleaseuseapplicationContext.collection()instead.

ForFoxxRepositoryobjects,thepropertymodelPrototypeisnowdeprecated.Usingitwillissueawarning.PleaseuseFoxxRepository.modelinstead.

InFoxxController/RequestContext,callingmethodbodyParam()withthreeargumentsisdeprecated.Pleaseuse.bodyParam(paramName,options)instead.

InFoxxController/RequestContextcallingmethodqueryParam({type:string})isdeprecated.PleaseuserequestContext.queryParam({type:joi})instead.

InFoxxController/RequestContextcallingmethodpathParam({type:string})isdeprecated.PleaseuserequestContext.pathParam({type:joi})instead.

ForFoxxModel,callingModel.extend({},{attributes:{}})isdeprecated.PleaseuseModel.extend({schema:{}})instead.

Inmoduleorg/arangodb/general-graph,thefunctions_undirectedRelation()and_directedRelation()aredeprecatedandwillberemovedinafutureversionofArangoDB.Bothfunctionshavebeenunifiedto_relation().

Themodulesorg/arangodb/graphandorg/arangodb/graph-blueprintaredeprecated.Pleaseusemoduleorg/arangodb/general-graphinstead.

TheHTTPRESTAPI_api/graphandallitsmethodsaredeprecated.PleaseusethegeneralgraphAPI_api/gharialinstead.

Removedfeatures

Thefollowingreplication-relatedJavaScriptmethodsbecameobsoleteinArangoDB2.2andhavebeenremovedinArangoDB2.4:

require("org/arangodb/replication").logger.start()

require("org/arangodb/replication").logger.stop()

require("org/arangodb/replication").logger.properties()

TheRESTAPImethodsforthesefunctionshavealsobeenremovedinArangoDB2.4:

HTTPPUT/_api/replication/logger-startHTTPPUT/_api/replication/logger-stopHTTPGET/_api/replication/logger-configHTTPPUT/_api/replication/logger-config

Clientapplicationsthatcalloneofthesemethodsshouldbeadjustedbyremovingthecallstothesemethods.Thisshouldn'tbeproblematicasthesemethodshavebeenno-opssinceArangoDB2.2anyway.


619


AQLimprovements

Frameworkimprovements

AQLqueriesarenowsentthroughaqueryoptimizerframeworkbeforeexecution.Thequeryoptimizerframeworkwillfirstconverttheinternalrepresentationofthequery,theabstractsyntaxtree,intoaninitialexecutionplan.

Theexecutionplanisthensendthroughoptimizerrulesthatmaydirectlymodifytheplaninplaceorcreateanewvariantoftheplan.Newplansmightagainbeoptimized,allowingtheoptimizertocarryoutseveraloptimizations.

Aftercreatingplans,theoptimizerwillestimatethecostsforeachplanandpicktheplanwiththelowestcost(termedtheoptimalplan)fortheactualqueryexecution.

Withtheexplain()methodofArangoStatementuserscancheckwhichexecutionplantheoptimizerpickorretrievealistofotherplansthatoptimizerdidnotchoose.Theplanwillrevealmanydetailsaboutwhichindexesareusedetc.explain()willalsoreturntheofoptimizerrulesappliedsouserscanvalidatewhetherornotaqueryallowsusingaspecificoptimization.

ExecutionofAQLquerieshasbeenrewritteninC++,allowingmanyqueriestoavoidtheconversionofdocumentsbetweenArangoDB'sinternallow-leveldatastructureandtheV8objectrepresentationformat.

Theframeworkforoptimizerrulesisnowalsogenerallycluster-aware,allowingspecificoptimizationsforqueriesthatruninacluster.Additionally,theoptimizerwasdesignedtobeextensibleinordertoaddmoreoptimizationsinthefuture.


Alternativeoperatorsyntax

ArangoDB2.3allowstousethefollowingalternativeformsforthelogicaloperators:

AND:logicalandOR:logicalorNOT:negation

Thisnewsyntaxisjustanalternativetotheoldsyntax,allowingeasiermigrationfromSQL.Theoldsyntaxisstillfullysupportedandwillbe:

&&:logicaland||:logicalor!:negation

NOTINoperator

AQLnowhasadedicatedNOTINoperator.

Previously,aNOTINwasonlyachievablebywritinganegatedINcondition:

FORiIN...FILTER!(iIN[23,42])...

InArangoDB2.3,thesameresultcannowalternativelybeachievedbywritingthemoreintuitivevariant:

FORiIN...FILTERiNOTIN[23,42]...

Improvementsofbuilt-infunctions

WhatsNewin2.3

620

ThefollowingAQLstringfunctionshavebeenadded:

LTRIM(value,characters):left-trimsastringvalueRTRIM(value,characters):right-trimsastringvalueFIND_FIRST(value,search,start,end):findsthefirstoccurrenceofasearchstringFIND_LAST(value,search,start,end):findsthelastoccurrenceofasearchstringSPLIT(value,separator,limit):splitsastringintoanarray,usingaseparatorSUBSTITUTE(value,search,replace,limit):replacescharactersorstringsinsideanother

ThefollowingotherAQLfunctionshavebeenadded:

VALUES(document):returnsthevaluesofanobjectasanarray(thisisthecounterparttothealreadyexistingATTRIBUTESfunction)ZIP(attributes,values):returnsanobjectconstructedfromattributesandvaluespassedinseparateparametersPERCENTILE(values,n,method):returnsthenthspercentileofthevaluesprovided,usingrankorinterpolationmethod

ThealreadyexistingfunctionsCONCATandCONCAT_SEPARATORnowsupportarrayarguments,e.g.:

/*"foobarbaz"*/

CONCAT(['foo','bar','baz'])

/*"foo,bar,baz"*/

CONCAT_SEPARATOR(",",['foo','bar','baz'])

AQLqueriesthrowlessexceptions

InpreviousversionsofArangoDB,AQLqueriesabortedwithanexceptioninmanysituationsandthrewaruntimeexception.Forexample,exceptionswerethrownwhentryingtofindavalueusingtheINoperatorinanon-arrayelement,whentryingtousenon-booleanvalueswiththelogicaloperands&&or||or!,whenusingnon-numericvaluesinarithmeticoperations,whenpassingwrongparametersintofunctionsetc.

ThefactthatmanyAQLoperatorscouldthrowexceptionsledtoalotofquestionsfromusers,andalotofmore-verbose-than-necessaryqueries.Forexample,thefollowingqueryfailedwhenthereweredocumentsthatdidnothaveatopicsattributeatall:

FORdocINmycollection

FILTERdoc.topicsIN["something","whatever"]

RETURNdoc

Thisforceduserstorewritethequeryasfollows:


FILTERIS_LIST(doc.topics)&&doc.topicsIN["something","whatever"]

RETURNdoc

InArangoDB2.3thishasbeenchangedtomakeAQLeasiertouse.Thechangeprovidesanextrabenefit,andthatisthatnon-throwingoperatorsallowthequeryoptimizertoperformmuchmoretransformationsinthequerywithoutchangingitsoverallresult.

Hereisasummaryofchanges:

whenanon-arrayvalueisusedontheright-handsideoftheINoperator,theresultwillbefalseinArangoDB2.3,andnoexceptionwillbethrown.thebooleanoperators&&and||donotthrowinArangoDB2.3ifanyoftheoperandsisnotabooleanvalue.Instead,theywillperformanimplicitcastofthevaluestobooleans.Theirresultwillbeasfollows:

lhs&&rhswillreturnlhsifitisfalseorwouldbefalsewhenconvertedintoaboolean.Iflhsistrueorwouldbetruewhenconvertedtoaboolean,rhswillbereturned.lhs||rhswillreturnlhsifitistrueorwouldbetruewhenconvertedintoaboolean.Iflhsisfalseorwouldbefalsewhenconvertedtoaboolean,rhswillbereturned.!valuewillreturnthenegatedvalueofvalueconvertedintoaboolean

thearithmeticoperators(+,-,*,/,%)canbeappliedtoanyvalueandwillnotthrowexceptionswhenappliedtonon-numericvalues.Instead,anyvalueusedintheseoperatorswillbecastedtoanumericvalueimplicitly.Ifnonumericresultcanbeproducedbyanarithmeticoperator,itwillreturnnullinArangoDB2.3.Thisisalsotruefordivisionbyzero.passingargumentsofinvalidtypesintoAQLfunctionsdoesnotthrowaruntimeexceptioninmostcases,butmayproduceruntime

WhatsNewin2.3

621

warnings.Built-inAQLfunctionsthatreceiveinvalidargumentswillthenreturnnull.

Performanceimprovements

Non-uniquehashindexes

Theperformanceofinsertionintonon-uniquehashindexeshasbeenimprovedsignificantly.Thisfixesperformanceproblemsincaseattributeswereindexesthatcontainedonlyveryfewdistinctvalues,orwhenmostofthedocumentsdidnotevencontaintheindexedattribute.Thisalsofixesproblemswhenloadingcollectionswithsuchindexes.

Theinsertiontimenowscaleslinearlywiththenumberofdocumentsregardlessofthecardinalityoftheindexedattribute.

Reverseiterationoverskiplistindexes

AQLqueriescannowuseasortedskiplistindexforreverseiteration.ThisallowsseveralqueriestorunfasterthaninpreviousversionsofArangoDB.

Forexample,thefollowingAQLquerycannowusetheindexondoc.value:


FILTERdoc.value>23

SORTdoc.valuesDESC

RETURNdoc

PreviousversionsofArangoDBdidnotusetheindexbecauseofthedescending(DESC)sort.

Additionally,thenewAQLoptimizercanuseanindexforsortingnoweveniftheAQLquerydoesnotcontainaFILTERstatement.ThisoptimizationwasnotavailableinpreviousversionsofArangoDB.

AddedbasicsupportforhandlingbinarydatainFoxx

BufferobjectscannowbeusedwhensettingtheresponsebodyofanyFoxxaction.ThisallowsFoxxactionstoreturnbinarydata.

RequestswithbinarypayloadcanbeprocessedinFoxxapplicationsbyusingthenewmethodres.rawBodyBuffer().ThiswillreturntheunparsedrequestbodyasaBufferobject.

Thereisnowalsothemethodreq.requestParts()availableinFoxxtoretrievetheindividualcomponentsofamultipartHTTPrequest.Thatcanbeusedforexampletoprocessfileuploads.

Additionally,theres.send()methodhasbeenaddedasaconveniencemethodforreturningstrings,JSONobjectsorBuffersfromaFoxxaction.Itprovidessomeauto-detectionbasedonitsparametervalue:

res.send("<p>someHTML</p>");//returnsanHTMLstring

res.send({success:true});//returnsaJSONobject

res.send(newBuffer("somebinarydata"));//returnsbinarydata

Theconveniencemethodres.sendFile()cannowbeusedtoreturnthecontentsofafilefromaFoxxaction.Theyfilemaycontainbinarydata:

res.sendFile(applicationContext.foxxFilename("image.png"));

Thefilesystemmethodsfs.write()andfs.readBuffer()canbeusedtoworkwithbinarydata,too:

fs.write()willperformanauto-detectionofitssecondparameter'svaluesoitworkswithBufferobjects:

fs.write(filename,"somedata");//savesastringvalueinfile

fs.write(filename,newBuffer("somebinarydata"));//saves(binary)contentsofabuffer

fs.readBuffer()hasbeenaddedasamethodtoreadthecontentsofanarbitraryfileintoaBufferobject.

WhatsNewin2.3

622

Webinterface

Batchdocumentremovalandmovefunctionalityhasbeenaddedtothewebinterface,makingiteasiertoworkwithmultipledocumentsatonce.Additionally,basicJSONimportandexporttoolshavebeenadded.

Command-lineoptionsadded

Thecommand-lineoption--javascript.v8-contextswasaddedtoarangodtoprovidebettercontroloverthenumberofV8contextscreatedinarangod.

Previously,thenumberofV8contextsarangodcreatedatstartupwasequaltothenumberofserverthreads(asspecifiedbyoption--server.threads).

InsomesituationsitmaybemoresensibletocreatedifferentamountsofthreadsandV8contexts.ThisisbecauseeachV8contextscreatedwillconsumememoryandrequiresCPUresourcesforperiodicgarbagecollection.Contrary,serverthreadsdonothavesuchhighmemoryorCPUfootprint.

Iftheoption--javascript.v8-contextsisnotspecified,thenumberofV8contextscreatedatstartupwillremainequaltothenumberofserverthreads.ThusnochangeinconfigurationisrequiredtokeepthesamebehaviorasinpreviousArangoDBversions.

Thecommand-lineoption--log.use-local-timewasaddedtoprintdatesandtimesinArangoDB'slogintheserver-localtimezoneinsteadofUTC.Ifitisnotset,thetimezonewilldefaulttoUTC.

Theoption--backslash-escapehasbeenaddedtoarangoimp.SpecifyingthisoptionwillusethebackslashastheescapecharacterforliteralquoteswhenparsingCSVfiles.Theescapecharacterforliteralquotesisstillthedoublequotecharacter.


ArangoDB'sbuilt-inHTTPservernowsupportsHTTPpipelining.

TheArangoShelltutorialfromthearangodb.comwebsiteisnowintegratedintotheArangoDBshell.

PowerfulFoxxEnhancementsWiththenewjobqueuefeatureyoucanrunasyncjobstocommunicatewithexternalservices,FoxxqueriesmakewritingcomplexAQLqueriesmucheasierandFoxxsessionswillhandletheauthenticationandsessionhassleforyou.

FoxxQueries

WritinglongAQLqueriesinJavaScriptcanquicklybecomeunwieldy.Asof2.3ArangoDBbundlestheArangoDBQueryBuildermodulethatprovidesaJavaScriptAPIforwritingcomplexAQLquerieswithoutstringconcatenation.Allbuilt-infunctionsthatacceptAQLstringsnowsupportquerybuilderinstancesdirectly.AdditionallyFoxxprovidesamethodFoxx.createQueryforcreatingparametrizedqueriesthatcanreturnFoxxmodelsorapplyarbitrarytransformationstothequeryresults.

FoxxSessions

ThesessionfunctionalityinFoxxhasbeencompletelyrewritten.TheoldactivateAuthenticationAPIisstillsupportedbutmaybedeprecatedinthefuture.ThenewactivateSessionsAPIsupportscookiesorconfigurableheaders,providesoptionalJSONWebTokenandcryptographicsigningsupportandusesthenewsessionsFoxxapp.

ArangoDB2.3providesFoxxappsforusermanagementandsaltedhash-basedauthenticationwhichcanbereplacedwithorsupplementedbyalternativeimplementations.Foranexampleappusingboththebuilt-inauthenticationandOAuth2seetheFoxxSessionsExampleapp.

FoxxQueues

FoxxnowprovidesasyncworkersviatheFoxxQueuesAPI.Jobsenqueuedinajobqueuewillbeexecutedasynchronouslyoutsideoftherequest/responsecycleofFoxxcontrollersandcanbeusedtocommunicatewithexternalservicesorperformtasksthattakealongtimetocompleteormayrequiremultipleattempts.

WhatsNewin2.3

623

https://npmjs.org/package/aqb

https://github.com/arangodb/foxx-sessions-example

Jobscanbescheduledinadvanceorsettobeexecutedimmediately,thenumberofretryattempts,theretrydelayaswellassuccessandfailurehandlerscanbedefinedforeachjobindividually.Jobtypesthatintegratevariousexternalservicesfortransactionale-mails,loggingandusertrackingcanbefoundintheFoxxappregistry.

Misc

TherequestandresponseobjectsinFoxxcontrollersnowprovidemethodsforreadingandwritingrawcookiesandsignedcookies.

MountedFoxxappswillnowbeloadedwhenarangodstartsratherthanatthefirstdatabaserequest.Thismayresultinslightlyslowerstartuptimes(butafasterresponseforthefirstrequest).

WhatsNewin2.3

624


Configurationfilechanges

Threadsandcontexts

Thenumberofserverthreadsspecifiedisnowtheminimumofthreadsstarted.Therearesituationinwhichthreadsarewaitingforresultsofdistributeddatabaseservers.Inthiscasethenumberofthreadsisdynamicallyincreased.

WithArangoDB2.3,thenumberofserverthreadscanbeconfiguredindependentlyofthenumberofV8contexts.Theconfigurationoption--javascript.v8-contextswasaddedtoarangodtoprovidebettercontroloverthenumberofV8contextscreatedinarangod.

Previously,thenumberofV8contextsarangodcreatedatstartupwasequaltothenumberofserverthreads(asspecifiedbyoption--server.threads).

InsomesituationsitmaybemoresensibletocreatedifferentamountsofthreadsandV8contexts.ThisisbecauseeachV8contextscreatedwillconsumememoryandrequiresCPUresourcesforperiodicgarbagecollection.Contrary,serverthreadsdonothavesuchhighmemoryorCPUfootprint.

Iftheoption--javascript.v8-contextsisnotspecified,thenumberofV8contextscreatedatstartupwillremainequaltothenumberofserverthreads.ThusnochangeinconfigurationisrequiredtokeepthesamebehaviorasinpreviousArangoDBversions.

Ifyouareusingthedefaultconfigfilesormergethemwithyourlocalconfigfiles,pleasereviewifthedefaultnumberofserverthreadsisokayinyourenvironment.AdditionallyyoushouldverifythatthenumberofV8contextscreated(asspecifiedinoption--javascript.v8-contexts)isokay.

Syslog

Thecommand-lineoption--log.syslogwasusedinpreviousversionsofArangoDBtoturnloggingtosyslogonoroff:whensettingtoanon-emptystring,syslogloggingwasturnedon,otherwiseturnedoff.Whensyslogloggingwasturnedon,loggingwasdonewiththeapplicationnamespecifiedin--log.application,whichdefaultedtotriagens.Therewasalsoacommand-lineoption--log.hostnamewhichcouldbesetbutdidnothaveanyeffect.

Thisbehaviorturnedouttobeunintuitiveandwaschangedin2.3asfollows:

thecommand-lineoption--log.syslogisdeprecatedanddoesnothaveanyeffectwhenstartingArangoDB.toturnonsyslogloggingin2.3,theoption--log.facilityhastobesettoanon-emptystring.ThevalueforfacilityisOS-dependent(possiblevaluescanbefoundin/usr/include/syslog.horthelike-usershouldbeavailableonmanysystems).thedefaultvaluefor--log.applicationhasbeenchangedfromtriagenstoarangod.thecommand-lineoption--log.hostnameisdeprecatedanddoesnothaveanyeffectwhenstartingArangoDB.Instead,thehostnamewillbesetbysyslogautomatically.whenloggingtosyslog,ArangoDBnowomitsthedatetimeprefixandtheprocessid,becausethey'llbeaddedbysyslogautomatically.

AQL

AQLqueriesthrowlessexceptions

ArangoDB2.3containsacompletelyrewrittenAQLqueryoptimizerandexecutionengine.ThismeansthatAQLquerieswillbeexecutedwithadifferentenginethaninArangoDB2.2andearlier.PartsofAQLqueriesmightbeexecutedindifferentorderthanbeforebecausetheAQLoptimizerhasmorefreedomtomovethingsaroundinaquery.


625

InpreviousversionsofArangoDB,AQLqueriesabortedwithanexceptioninmanysituationsandthrewaruntimeexception.ExceptionswerethrownwhentryingtofindavalueusingtheINoperatorinanon-arrayelement,whentryingtousenon-booleanvalueswiththelogicaloperands&&or||or!,whenusingnon-numericvaluesinarithmeticoperations,whenpassingwrongparametersintofunctionsetc.

InArangoDB2.3thishasbeenchangedinmanycasestomakeAQLmoreuser-friendlyandtoallowtheoptimizationtoperformmuchmorequeryoptimizations.

Hereisasummaryofchanges:

whenanon-arrayvalueisusedontheright-handsideoftheINoperator,theresultwillbefalseinArangoDB2.3,andnoexceptionwillbethrown.thebooleanoperators&&and||donotthrowinArangoDB2.3ifanyoftheoperandsisnotabooleanvalue.Instead,theywillperformanimplicitcastofthevaluestobooleans.Theirresultwillbeasfollows:

lhs&&rhswillreturnlhsifitisfalseorwouldbefalsewhenconvertedintoaboolean.Iflhsistrueorwouldbetruewhenconvertedtoaboolean,rhswillbereturned.lhs||rhswillreturnlhsifitistrueorwouldbetruewhenconvertedintoaboolean.Iflhsisfalseorwouldbefalsewhenconvertedtoaboolean,rhswillbereturned.!valuewillreturnthenegatedvalueofvalueconvertedintoaboolean

thearithmeticoperators(+,-,*,/,%)canbeappliedtoanyvalueandwillnotthrowexceptionswhenappliedtonon-numericvalues.Instead,anyvalueusedintheseoperatorswillbecastedtoanumericvalueimplicitly.Ifnonumericresultcanbeproducedbyanarithmeticoperator,itwillreturnnullinArangoDB2.3.Thisisalsotruefordivisionbyzero.passingargumentsofinvalidtypesintoAQLfunctionsdoesnotthrowaruntimeexceptioninmostcases,butmayproduceruntimewarnings.Built-inAQLfunctionsthatreceiveinvalidargumentswillthenreturnnull.

NestedFORloopexecutionorder

Thequeryoptimizerin2.3maypermutetheorderofnestedFORloopsinAQLqueries,providedthatexchangingtheloopswillnotalteraqueryresult.However,achangeintheorderofreturnedvaluesisallowedbecausenosortorderisguaranteedbyAQL(andwasnever)unlessanexplicitSORTstatementisusedinaquery.

ChangedreturnvaluesofArangoQueryCursor.getExtra()

ThereturnvalueofArangoQueryCursor.getExtra()hasbeenchangedinArangoDB2.3.Itnowcontainsastatsattributewithstatisticsaboutthequerypreviouslyexecuted.Italsocontainsawarningsattributewithwarningsthathappenedduringqueryexecution.Thereturnvaluestructurehasbeenunifiedin2.3forbothread-onlyanddata-modificationqueries.

Thereturnvaluelookslikethisforaread-onlyquery:

arangosh>stmt=db._createStatement("FORiINmycollectionRETURNi");stmt.execute().getExtra()

{

"stats":{

"writesExecuted":0,

"writesIgnored":0,

"scannedFull":2600,

"scannedIndex":0

},

"warnings":[]

}

Fordata-modificationqueries,ArangoDB2.3returnsaresultwiththesamestructure:

arangosh>stmt=db._createStatement("FORiINxxREMOVEiINxx");stmt.execute().getExtra()

{

"stats":{

"writesExecuted":2600,

"writesIgnored":0,

"scannedFull":2600,

"scannedIndex":0

},

"warnings":[]

}


626

InArangoDB2.2,thereturnvalueofArangoQueryCursor.getExtra()wasemptyforread-onlyqueriesandcontainedanattributeoperationswithtwosub-attributesfordata-modificationqueries:

arangosh>stmt=db._createStatement("FORiINmycollectionRETURNi");stmt.execute().getExtra()

{

}

arangosh>stmt=db._createStatement("FORiINmycollectionREMOVEiINmycollection");stmt.execute().getExtra()

{

"operations":{

"executed":2600,

"ignored":0

}

}

ChangedreturnvaluesinHTTPmethodPOST/_api/cursor

ThepreviouslymentionedchangealsoleadstothestatisticsbeingreturnedintheHTTPRESTAPImethodPOST/_api/cursor.Previously,thereturnvaluecontainedanoptionalextraattributethatwasfilledonlyfordata-modificationqueriesandinsomeothercasesasfollows:

{

"result":[],

"hasMore":false,

"extra":{

"operations":{

"executed":2600,

"ignored":0

}

}

}

WiththechangedresultstructureinArangoDB2.3,theextraattributeintheresultwilllooklikethis:

{

"result":[],

"hasMore":false,

"extra":{

"stats":{

"writesExecuted":2600,

"writesIgnored":0,

"scannedFull":0,

"scannedIndex":0

},

"warnings":[]

}

}

IfthequeryoptionfullCountisrequested,thefullCountresultvaluewillalsobereturnedinsidethestatsattributeoftheextraattribute,andnotdirectlyasanattributeinsidetheextraattributeasin2.2.NotethatafullCountwillonlybepresentinextra.statsifitwasrequestedasanoptionforthequery.

TheresultinArangoDB2.3willalsocontainawarningsattributewiththearrayofwarningsthathappenedduringqueryexecution.

ChangedreturnvaluesinArangoStatement.explain()

ThereturnvalueofArangoStatement.explain()haschangedsignificantlyinArangoDB2.3.Thenewreturnvaluestructureisnotcompatiblewiththestructurereturnedby2.2.

InArangoDB2.3,thefullexecutionplanforanAQLqueryisreturnedalongsideallappliedoptimizerrules,optimizationwarningsetc.Itisalsopossibletohavetheoptimizerreturnallexecutionplans.Thisrequiredanewdatastructure.

ClientprogramsthatuseArangoStatement.explain()ortheHTTPRESTAPImethodPOST/_api/explainmayneedtobeadjustedtousethenewreturnformat.


627

ThereturnvalueofArangoStatement.parse()hasbeenextendedinArangoDB2.3.Inadditiontotheexistingattributes,ArangoDB2.3willalsoreturnanastattributecontainingtheabstractsyntaxtreeofthestatement.Thisextraattributecansafelybeignoredbyclientprograms.

Variablesnotupdatableinqueries

PreviousversionsofArangoDBallowedthemodificationofvariablesinsideAQLqueries,e.g.

LETcounter=0

FORiIN1..10

LETcounter=counter+1

RETURNcounter

Whilethisisadmittedlyaconvenientfeature,thenewqueryoptimizerdesigndidnotallowtokeepit.Additionally,updatingvariablesinsideaquerywouldpreventalotofoptimizationstoqueriesthatwewouldliketheoptimizertomake.Additionally,updatingvariablesinqueriesthatrunondifferentnodesinaclusterwouldlikecausenon-deterministicbehaviorbecausequeriesarenotexecutedlinearly.

ChangedreturnvalueofTO_BOOL

TheAQLfunctionTO_BOOLnowalwaysreturnstrueifitsargumentisanarrayoranobject.InpreviousversionsofArangoDB,thefunctionreturnedfalseforemptyarraysorforobjectswithoutattributes.

ChangedreturnvalueofTO_NUMBER

TheAQLfunctionTO_NUMBERnowreturnsnullifitsargumentisanobjectoranarraywithmorethanonemember.InpreviousversionofArangoDB,thereturnvalueinthesecaseswas0.TO_NUMBERwillreturn0foremptyarray,andthenumericequivalentofthearraymember'svalueforarrayswithasinglemember.

NewAQLkeywords

ThefollowingkeywordshavebeenaddedtoAQLinArangoDB2.3:

NOTANDOR

UnquotedusageofthesekeywordsforattributenamesinAQLquerieswilllikelyfailinArangoDB2.3.Ifanysuchattributenameneedstobeusedinaquery,itshouldbeenclosedinbacktickstoindicatetheusageofaliteralattributename.

Removedfeatures

Bitarrayindexes

Bitarrayindexeswereonlyhalf-waydocumentedandintegratedinpreviousversionsofArangoDBsotheirbenefitwaslimited.ThesupportforbitarrayindexeshasthusbeenremovedinArangoDB2.3.Itisnotpossibletocreateindexesoftype"bitarray"withArangoDB2.3.

WhenacollectionisopenedthatcontainsabitarrayindexdefinitioncreatedwithapreviousversionofArangoDB,ArangoDBwillignoreitandlogthefollowingwarning:

indextype'bitarray'isnotsupportedinthisversionofArangoDBandisignored

FutureversionsofArangoDBmayautomaticallyremovesuchindexdefinitionssothewarningswilleventuallydisappear.

Otherremovedfeatures

TheHTTPRESTAPImethodatPOST/_admin/modules/flushhasbeenremoved.


628

Knownissues

InArangoDB2.3.0,AQLqueriescontainingfilterconditionswithanINexpressionwillnotyetuseanindex:

FORdocINcollectionFILTERdoc.indexedAttributeIN[...]RETURNdoc

FORdocINcollection

FILTERdoc.indexedAttributeIN[...]

RETURNdoc

We’recurrentlyworkingongettingtheINoptimizationsdone,andwillshipthemina2.3maintenancereleasesoon(e.g.2.3.1or2.3.2).


629


AQLimprovements

DatamodificationAQLqueries

Uptoincludingversion2.1,AQLsupporteddataretrievaloperationsonly.StartingwithArangoDBversion2.2,AQLalsosupportsthefollowingdatamodificationoperations:

INSERT:insertnewdocumentsintoacollectionUPDATE:partiallyupdateexistingdocumentsinacollectionREPLACE:completelyreplaceexistingdocumentsinacollectionREMOVE:removeexistingdocumentsfromacollection

Data-modificationoperationsarenormallycombinedwithotherAQLstatementssuchasFORloopsandFILTERconditionstodeterminethesetofdocumentstooperateon.Forexample,thefollowingquerywillfindalldocumentsincollectionusersthatmatchaspecificconditionandsettheirstatusvariabletoinactive:

FORuINusers

FILTERu.status=='notactive'

UPDATEuWITH{status:'inactive'}INusers

Thefollowingquerycopiesalldocumentsfromcollectionusersintocollectionbackup:

FORuINusers

INSERTuINbackup

Andthisqueryremovesdocumentsfromcollectionbackup:

FORdocINbackup

FILTERdoc.lastModified<DATE_NOW()-3600

REMOVEdocINbackup

Formoreinformationondata-modificationqueries,pleaserefertoDatamodificationqueries.

Updatablevariables

Previously,thevalueofavariableassignedinanAQLquerywiththeLETkeywordwasnotupdatableinanAQLquery.Thispreventedstatementslikethefollowingfrombeingexecutable:

LETsum=0

FORvINvalues

SORTv.year

LETsum=sum+v.value

RETURN{year:v.year,value:v.value,sum:sum}

OtherAQLimprovements

addedAQLTRANSLATEfunction

Thisfunctioncanbeusedtoperformlookupsfromstaticobjects,e.g.

LETcountryNames={US:"UnitedStates",UK:"UnitedKingdom",FR:"France"}

RETURNTRANSLATE("FR",countryNames)

WhatsNewin2.2

630

LETlookup={foo:"foo-replacement",bar:"bar-replacement",baz:"baz-replacement"}

RETURNTRANSLATE("foobar",lookup,"notcontained!")

Write-aheadlogAllwriteoperationsinanArangoDBserverwillnowbeautomaticallyloggedintheserver'swrite-aheadlog.Thewrite-aheadlogisasetofappend-onlylogfiles,anditisusedincaseofacrashrecoveryandforreplication.

Datafromthewrite-aheadlogwilleventuallybemovedintothejournalsordatafilesofcollections,allowingtheservertoremoveolderwrite-aheadlogfiles.

Cross-collectiontransactionsinArangoDBshouldbenefitconsiderablybythischange,aslesswritesthaninpreviousversionsarerequiredtoensurethedataofmultiplecollectionsareatomicallyanddurablycommitted.Alldata-modifyingoperationsinsidetransactions(insert,update,remove)willwritetheiroperationsintothewrite-aheadlogdirectlynow.Inpreviousversions,suchoperationswerebuffereduntilthecommitorrollbackoccurred.TransactionswithmultipleoperationsshouldthereforerequirelessphysicalmemorythaninpreviousversionsofArangoDB.

Thedatainthewrite-aheadlogcanalsobeusedinthereplicationcontext.InpreviousversionsofArangoDB,replicatingfromamasterrequiredturningonaspecialreplicationloggeronthemaster.Thereplicationloggercausedanextrawriteoperationintothe_replicationsystemcollectionforeachactualwriteoperation.Thisextrawriteisnowsuperfluous.Instead,slavescanreaddirectlyfromthemaster'swrite-aheadlogtogetinformedaboutmostrecentdatachanges.Thisremovestheneedtostoredata-modificationoperationsinthe_replicationcollectionaltogether.

Fortheconfigurationofthewrite-aheadlog,pleaserefertoWrite-aheadlogoptions.

Theintroductionofthewrite-aheadlogalsoremovestheneedtoconfigureandstartthereplicationloggeronamaster.ThoughthereplicationloggerobjectisstillavailableinArangoDB2.2toensureAPIcompatibility,starting,stopping,orconfiguringitwillhavenoeffect.

PerformanceimprovementsRemovedsortingofattributenameswhenincollectionshaper

InpreviousversionsofArangoDB,addingadocumentwithpreviouslynot-usedattributenamescausedafullsortofallattributenamesusedinthecollection.Thesortingwasdonetoensurefastcomparisonsofattributenamesinsomerareedgecases,butitconsiderablysloweddowninsertsintocollectionswithmanydifferentorevenuniqueattributenames.

Specializedprimaryindeximplementationtoallowfasterhashtablerebuildingandreducelookupsindatafilesfortheactualvalueof_key.Thisalsoreducestheamountofrandommemoryaccessesforprimaryindexinserts.

Reclamationofindexmemorywhendeletinglastdocumentincollection

Deletingdocumentsfromacollectiondidnotleadtoindexsizesbeingreduced.Instead,theindexmemorywaskeptallocatedandre-usedlaterwhenacollectionwasrefilledwithnewdocuments.Now,indexmemoryofprimaryindexesandhashindexesisreclaimedinstantlywhenthelastdocumentinacollectionisremoved.

Preventbufferingoflongprintresultsinarangosh'sandarangod'sprintcommand

Thischangewillemitbufferedintermediateprintresultsanddiscardtheoutputbuffertoquicklydeliverprintresultstotheuser,andtopreventconstructingverylargebuffersforlargeresults.


Addedinsertmethodasanaliasforsave.Documentscannowbeinsertedintoacollectionusingeithermethod:

db.test.save({foo:"bar"});

db.test.insert({foo:"bar"});

Cleanupofoptionsfordata-modificationoperations

WhatsNewin2.2

631

Manyofthedata-modificationoperationshadsignatureswithmanyoptionalboolparameters,e.g.:

db.test.update("foo",{bar:"baz"},true,true,true)

db.test.replace("foo",{bar:"baz"},true,true)

db.test.remove("foo",true,true)

db.test.save({bar:"baz"},true)

Suchlongparameterlistswereunintuitiveandhardtousewhenonlyoneoftheoptionalparametersshouldhavebeenset.

TomaketheAPIsmoreusable,theoperationsnowunderstandthefollowingalternativesignature:

collection.update(key,update-document,options)

collection.replace(key,replacement-document,options)

collection.remove(key,options)

collection.save(document,options)

Examples:

db.test.update("foo",{bar:"baz"},{overwrite:true,keepNull:true,waitForSync:true})

db.test.replace("foo",{bar:"baz"},{overwrite:true,waitForSync:true})

db.test.remove("foo",{overwrite:true,waitForSync:true})

db.test.save({bar:"baz"},{waitForSync:true})

Added--overwriteoptiontoarangoimp

Thisallowsremovingalldocumentsinacollectionbeforeimportingintoitusingarangoimp.

Honorstartupoption--server.disable-statisticswhendecidingwhetherornottostartperiodicstatisticscollectionjobs

Previously,thestatisticscollectionjobswerestartedeveniftheserverwasstartedwiththe--server.disable-statisticsflagbeingsettotrue.Nowiftheoptionissettotrue,nostatisticswillbecollectedontheserver.

DisallowstoringofJavaScriptobjectsthatcontainJavaScriptnativeobjectsoftypeDate,Function,RegExporExternal,e.g.

db.test.save({foo:/bar/});

db.test.save({foo:newDate()});

Thiswillnowprint

Error:<data>cannotbeconvertedintoJSONshape:couldnotshapedocument

Previously,objectsofthesetypesweresilentlyconvertedintoanemptyobject(i.e.{})andnowarningwasissued.

Tostoresuchobjectsinacollection,explicitlyconvertthemintostringslikethis:

db.test.save({foo:String(/bar/)});

db.test.save({foo:String(newDate())});

Removedfeatures

MRubyintegrationforarangod

ArangoDBhadanexperimentalMRubyintegrationinsomeofthepublishbuilds.Thiswasn'tcontinuouslydeveloped,andsoithasbeenremovedinArangoDB2.2.

Thischangehasledtothefollowingstartupoptionsbeingsuperfluous:

--ruby.gc-interval

--ruby.action-directory

--ruby.modules-path

--ruby.startup-directory

WhatsNewin2.2

632

SpecifyingthesestartupoptionswilldonothinginArangoDB2.2,sousingtheseoptionsshouldbeavoidedfromnowonastheymightberemovedinafutureversionofArangoDB.

Removedstartupoptions

ThefollowingstartupoptionshavebeenremovedinArangoDB2.2.Specifyingthemintheserver'sconfigurationfilewillnotproduceanerrortomakemigrationeasier.Still,usageoftheseoptionsshouldbeavoidedastheywillnothaveanyeffectandmightfullyberemovedinafutureversionofArangoDB:

--database.remove-on-drop

--database.force-sync-properties

--random.no-seed

--ruby.gc-interval

--ruby.action-directory

--ruby.modules-path

--ruby.startup-directory

--server.disable-replication-logger

MultiCollectionGraphs

ArangoDBisamultimodeldatabasewithnativegraphsupport.Inversion2.2thefeaturesforgraphshavebeenimprovedbyintegrationofanewgraphmodule.Allgraphscreatedwiththeoldmoduleareautomaticallymigratedintothenewmodulebutcanstillbeusedbytheoldmodule.

Newgraphmodule

Uptoincludingversion2.1,ArangoDBofferedamoduleforgraphsandgraphoperations.Thismoduleallowedyoutouseexactlyoneedgecollectiontogetherwithonevertexcollectioninagraph.WithArangoDBversion2.2thisgraphmoduleisdeprecatedandanewgraphmoduleisoffered.Thisnewmoduleallowstocombineanarbitrarynumberofvertexcollectionsandedgecollectionsinthesamegraph.Foreachedgecollectionalistofcollectionscontainingsourceverticesandalistofcollectionscontainingtargetverticescanbedefined.IfanedgeisstoredArangoDBchecksifthisedgeisvalidinthiscollection.Furthermoreifavertexisremovedfromoneofthecollectionsallconnectededgeswillberemovedaswell,givingtheguaranteeofnolooseendsinthegraphs.Thelayoutofthegraphcanbemodifiedatruntimebyaddingorremovingcollectionsandchangingthedefinitionsforedgecollections.Alloperationsonthegraphlevelaretransactionalbydefault.

GraphsinAQL

MulticollectiongraphshavebeenaddedtoAQLaswell.Basicfunctionality(gettingvertices,edges,neighbors)canbeexecutedusingtheentiregraph.Alsomoreadvancedfeatureslikeshortestpathcalculations,characteristicfactorsofthegraphortraversalshavebeenintegratedintoAQL.Forthesefunctionsallgraphscreatedwiththegraphmodulecanbeused.

WhatsNewin2.2

633


NewEdgesIndex

Theedgesindex(usedtostoreconnectionsbetweennodesinagraph)internallyusesanewdatastructure.Thisdatastructureimprovestheperformancewhenpopulatingtheedgeindex(i.e.whenloadinganedgecollection).Forlargegraphsloadingcanbe20timesfasterthanwithArangoDB2.0.

Additionally,thenewindexfixesperformanceproblemsthatoccurredwhenmanyduplicate_fromor_tovalueswerecontainedintheindex.Furthermore,thenewindexsupportsfasterremovalofedges.

Finally,whenloadinganexistingcollectionandbuildingtheedgesindexforthecollection,lessmemoryre-allocationswillbeperformed.

Overall,thisshouldconsiderablyspeeduploadingedgecollections.

Thenewindextypereplacestheoldedgesindextypeautomatically,withoutanychangesbeingrequiredbytheenduser.

TheAPIofthenewindexiscompatiblewiththeAPIoftheoldindex.Stillitispossiblethatthenewindexreturnsedgesinadifferentorderthantheoldindex.Thisisstillconsideredtobecompatiblebecausetheoldindexhadneverguaranteedanyresultordereither.

AQLImprovements

AQLoffersfunctionalitytoworkwithdates.DatesarenodatatypesoftheirowninAQL(neithertheyareinJSON,whichisoftenusedasaformattoshipdataintoandoutofArangoDB).Instead,datesinAQLareinternallyrepresentedbyeithernumbers(timestamps)orstrings.ThedatefunctionsinAQLprovidemechanismstoconvertfromanumerictimestamptoastringrepresentationandviceversa.

TherearetwodatefunctionsinAQLtocreatedatesforfurtheruse:

DATE_TIMESTAMP(date)CreatesaUTCtimestampvaluefromdate

DATE_TIMESTAMP(year,month,day,hour,minute,second,millisecond):Sameasbefore,butallowsspecifyingtheindividualdatecomponentsseparately.Allparametersafterdayareoptional.

DATE_ISO8601(date):ReturnsanISO8601datetimestringfromdate.ThedatetimestringwillalwaysuseUTCtime,indicatedbytheZatitsend.

DATE_ISO8601(year,month,day,hour,minute,second,millisecond):sameasbefore,butallowsspecifyingtheindividualdatecomponentsseparately.Allparametersafterdayareoptional.

Thesetwoabovedatefunctionsacceptthefollowinginputvalues:

numerictimestamps,indicatingthenumberofmillisecondselapsedsincetheUNIXepoch(i.e.January1st197000:00:00UTC).Anexampletimestampvalueis1399472349522,whichtranslatesto2014-05-07T14:19:09.522Z.

datetimestringsinformatsYYYY-MM-DDTHH:MM:SS.MMM,YYYY-MM-DDHH:MM:SS.MMM,orYYYY-MM-DD.Millisecondsarealwaysoptional.

Atimezonedifferencemayoptionallybeaddedattheendofthestring,withthehoursandminutesthatneedtobeaddedorsubtractedtothedatetimevalue.Forexample,2014-05-07T14:19:09+01:00canbeusedtospecifyaonehouroffset,and2014-05-07T14:19:09+07:30canbespecifiedforsevenandhalfhoursoffset.Negativeoffsetsarealsopossible.Alternativelytoanoffset,aZcanbeusedtoindicateUTC/Zulutime.

Anexamplevalueis2014-05-07T14:19:09.522ZmeaningMay7th2014,14:19:09and522milliseconds,UTC/Zulutime.Anotherexamplevaluewithouttimecomponentis2014-05-07Z.

Pleasenotethatifnotimezoneoffsetisspecifiedinadatestring,ArangoDBwillassumeUTCtimeautomatically.Thisisdonetoensureportabilityofqueriesacrossserverswithdifferenttimezonesettings,andbecausetimestampswillalwaysbeUTC-based.

WhatsNewin2.1

634

individualdatecomponentsasseparatefunctionarguments,inthefollowingorder:

yearmonthdayhourminutesecondmillisecond

Allcomponentsfollowingdayareoptionalandcanbeomitted.Notethatnotimezoneoffsetscanbespecifiedwhenusingseparatedatecomponents,andUTC/Zulutimewillbeused.

ThefollowingcallstoDATE_TIMESTAMPareequivalentandwillallreturn1399472349522:

DATE_TIMESTAMP("2014-05-07T14:19:09.522")

DATE_TIMESTAMP("2014-05-07T14:19:09.522Z")

DATE_TIMESTAMP("2014-05-0714:19:09.522")

DATE_TIMESTAMP("2014-05-0714:19:09.522Z")

DATE_TIMESTAMP(2014,5,7,14,19,9,522)

DATE_TIMESTAMP(1399472349522)

ThesameistrueforcallstoDATE_ISO8601thatalsoacceptsvariableinputformats:

DATE_ISO8601("2014-05-07T14:19:09.522Z")

DATE_ISO8601("2014-05-0714:19:09.522Z")

DATE_ISO8601(2014,5,7,14,19,9,522)

DATE_ISO8601(1399472349522)

Theabovefunctionsareallequivalentandwillreturn"2014-05-07T14:19:09.522Z".

ThefollowingdatefunctionscanbeusedwithdatescreatedbyDATE_TIMESTAMPandDATE_ISO8601:

DATE_DAYOFWEEK(date):Returnstheweekdaynumberofdate.Thereturnvalueshavethefollowingmeanings:

0:Sunday1:Monday2:Tuesday3:Wednesday4:Thursday5:Friday6:Saturday

DATE_YEAR(date):Returnstheyearpartofdateasanumber.

DATE_MONTH(date):Returnsthemonthpartofdateasanumber.

DATE_DAY(date):Returnsthedaypartofdateasanumber.

DATE_HOUR(date):Returnsthehourpartofdateasanumber.

DATE_MINUTE(date):Returnstheminutepartofdateasanumber.

DATE_SECOND(date):Returnsthesecondspartofdateasanumber.

DATE_MILLISECOND(date):Returnsthemillisecondspartofdateasanumber.

Thefollowingotherdatefunctionsarealsoavailable:

DATE_NOW():Returnsthecurrenttimeasatimestamp.

Notethatthisfunctionisevaluatedoneveryinvocationandmayreturndifferentvalueswheninvokedmultipletimesinthesamequery.

ThefollowingotherAQLfunctionshavebeenaddedinArangoDB2.1:

FLATTEN:thisfunctioncanturnanarrayofsub-arraysintoasingleflatarray.Allarrayelementsintheoriginalarraywillbeexpandedrecursivelyuptoaconfigurabledepth.Theexpandedvalueswillbeaddedtothesingleresultarray.

WhatsNewin2.1

635

Example:

FLATTEN([1,2,[3,4],5,[6,7],[8,[9,10]])

willexpandthesub-arraysonthefirstlevelandproduce:

[1,2,3,4,5,6,7,8,[9,10]]

Tofullyflattenthearray,themaximumdepthcanbespecified(e.g.withavalueof2):

FLATTEN([1,2,[3,4],5,[6,7],[8,[9,10]],2)

Thiswillfullyexpandthesub-arraysandproduce:

[1,2,3,4,5,6,7,8,9,10]

CURRENT_DATABASE:thisfunctionwillreturnthenameofthedatabasethecurrentqueryisexecutedin.

CURRENT_USER:thisfunctionreturnsthenameofthecurrentuserthatisexecutingthequery.Ifauthorizationisturnedofforthequeryisexecutedoutsideofarequestcontext,nouserispresentandthefunctionwillreturnnull.

ClusterDumpandRestore

Thedumpandrestoretools,arangodumpandarangorestore,cannowbeusedtodumpandrestorecollectionsinacluster.Additionally,acollectiondumpfromastandaloneArangoDBservercanbeimportedintoacluster,andviceversa.

WebInterfaceImprovements

Thewebinterfaceinversion2.1hasamorecompactdashboard.Itprovideschartswithtime-seriesforincomingrequests,HTTPtransfervolumeandsomeserverresourceusagefigures.

Additionallyitprovidestrendindicators(e.g.15minaverages)anddistributioncharts(akahistogram)forsomefigures.

FoxxImprovements

ToeasilyaccessafileinsidethedirectoryofaFoxxapplicationfromwithinFoxx,Foxx'sapplicationContextnowprovidesthefoxxFilename()function.Itcanbeusedtoassemblethefullfilenameofafileinsidetheapplication'sdirectory.TheapplicationContextcanbeaccessedasglobalvariablefromanymodulewithinaFoxxapplication.

ThefilenamecanbeusedinsideFoxxactionsorsetup/teardownscripts,e.g.topopulateaFoxxapplication'scollectionwithdata.

TherequirefunctionnowalsopreferslocalmoduleswhenusedfrominsideaFoxxapplication.ThisallowsputtingmodulesinsidetheFoxxapplicationdirectoryandrequiringthemeasily.Italsoallowsusingapplication-specificversionsoflibrariesthatarebundledwithArangoDB(suchasunderscore.js).

WindowsInstaller

TheWindowsinstallershippedwithArangoDBnowsupportsinstallationofArangoDBforthecurrentuserorallusers,withtherequiredprivileges.ItalsosupportstheinstallationofArangoDBasaservice.

Fixesfor32bitsystems

SeveralissueshavebeenfixedthatoccurredonlywhenusingArangoDBona32bitsoperatingsystem,specifically:

acrashinathirdpartycomponentusedtomanageclusterdata

WhatsNewin2.1

636

athirdpartylibrarythatfailedtoinitializeon32bitWindows,makingarangodandarangoshcrashimmediately.

overflowsofvaluesusedfornanosecond-precisiontimeouts:theseoverflowshaveledtoinvalidvaluesbeingpassedtosocketoperations,makingthemfailandre-trytoooften

Updateddrivers

SeveraldriversforArangoDBhavebeencheckedforcompatibilitywith2.1.Thecurrentlistofdriverswithcompatibilitynotescanbefoundonlinehere.

C++11usage

WehavemovedseveralfilesfromCtoC++,allowingmorecodereuseandreducingtheneedforshippingdatabetweenthetwo.WehavealsodecidedtorequireC++11supportforArangoDB,whichallowsustousesomeofthesimplifications,featuresandguaranteesthatthisstandardhasinstock.

ThatalsomeansacompilerwithC++11supportisrequiredtobuildArangoDBfromsource.ForinstanceGNUCCofatleastversion4.8.

MiscellaneousImprovementsCancelableasynchronousjobs:severalpotentiallylong-runningjobscannowbecanceledviaanexplicitcanceloperation.Thisallowsstoppinglong-runningqueries,traversalsorscriptswithoutshuttingdownthecompleteArangoDBprocess.Jobcancelationisprovidedforasynchronouslyexecutedjobsasisdescribedin@refHttpJobCancel.

Server-sideperiodictaskmanagement:anArangoDBservernowprovidesfunctionalitytoregisterandunregisterperiodictasks.Tasksareuser-definedJavaScriptactionsthatcanberunperiodicallyandautomatically,independentofanyHTTPrequests.

Thefollowingtaskmanagementfunctionsareprovided:

require("org/arangodb/tasks").register():registersaperiodictaskrequire("org/arangodb/tasks").unregister():unregistersandremovesaperiodictaskrequire("org/arangodb/tasks").get():retrievesaspecifictasksorallexistingtasks

Anexampletask(tobeexecutedevery15seconds)canberegisteredlikethis:

vartasks=require("org/arangodb/tasks");

tasks.register({

name:"thisisanexampletaskwithparameters",

period:15,

command:function(params){

vargreeting=params.greeting;

vardata=JSON.stringify(params.data);

require('console').log('%sfromparametertask:%s',greeting,data);

},

params:{greeting:"hi",data:"howareyou?"}

});

Pleaserefertothesection@refTasksformoredetails.

Thefiguresmethodofacollectionnowreturnsdataaboutthecollection'sindexmemoryconsumption.Thereturnedvalueindexes.sizewillcontainthetotalamountofmemoryacquiredbyallindexesofthecollection.Thisfigurecanbeusedtoassessthememoryimpactofindexes.

CapitalizedHTTPresponseheaders:fromversion2.1,ArangoDBwillreturncapitalizedHTTPheadersbydefault,e.g.Content-Lengthinsteadofcontent-length.ThoughtheHTTPspecificationstatesthatheadersfieldnamearecase-insensitive,severalolderclienttoolsrelyonaspecificcaseinHTTPresponseheaders.ThischangesmakeArangoDBabitmorecompatiblewiththose.

Simplifiedusageofdb._createStatement():toeasilyrunanAQLquery,themethoddb._createStatementnowallowspassingtheAQLqueryasastring.Previouslyitrequiredtheusertopassanobjectwithaqueryattribute(whichthencontainedthequerystring).

WhatsNewin2.1

637

https://www.arangodb.org/driver

ArangoDBnowsupportsbothversions:

db._createStatement(queryString);

db._createStatement({query:queryString});

WhatsNewin2.1

638

AppendixReferences:Briefoverviewsoverinterfacesandobjects

db:thedbobjectcollection:thecollectionobject

JavaScriptModules:Listofbuilt-inandsupportedJSmodulesDeprecated:FeaturesthatareconsideredobsoleteandmaygetremovedeventuallyErrorcodesandmeanings:ListofallpossibleerrorsthatcanbeencounteredGlossary:Disambiguationpage

Appendix

639

References

References

640

The"db"ObjectThedbobjectisavailableinarangoshbydefault,andcanalsobeimportedandusedinFoxxservices.

db.namereturnsacollectionobjectforthecollectionname.

Thefollowingmethodsexistsonthe_dbobject:

Database

db._createDatabase(name,options,users)db._databases()db._dropDatabase(name,options,users)db._useDatabase(name)

Indexes

db._index(index)db._dropIndex(index)

Properties

db._id()db._isSystem()db._name()db._path()db._version()

Collection

db._collection(name)db._create(name)db._drop(name)db._truncate(name)

AQL

db._createStatement(query)db._query(query)db._explain(query)

Document

db._document(object)db._exists(object)db._remove(selector)db._replace(selector,data)db._update(selector,data)

db

641

The"collection"ObjectThefollowingmethodsexistonthecollectionobject(returnedbydb.name):

Collection

collection.checksum()collection.count()collection.drop()collection.figures()collection.load()collection.properties()collection.revision()collection.rotate()collection.toArray()collection.truncate()collection.type()collection.unload()

Indexes

collection.dropIndex(index)collection.ensureIndex(description)collection.getIndexes(name)collection.index(index)

Document

collection.all()collection.any()collection.closedRange(attribute,left,right)collection.document(object)collection.documents(keys)collection.edges(vertex-id)collection.exists(object)collection.firstExample(example)collection.inEdges(vertex-id)collection.insert(data)collection.edges(vertices)collection.iterate(iterator,options)collection.outEdges(vertex-id)collection.queryByExample(example)collection.range(attribute,left,right)collection.remove(selector)collection.removeByKeys(keys)collection.rename()collection.replace(selector,data)collection.replaceByExample(example,data)collection.save(data)collection.update(selector,data)collection.updateByExample(example,data)

collection

642

JavaScriptModulesArangoDBusesaNode.jscompatiblemodulesystem.Youcanusethefunctionrequireinordertoloadamoduleorlibrary.Itreturnstheexportedvariablesandfunctionsofthemodule.

Theglobalvariablesglobal,process,console,Buffer,__filenameand__dirnameareavailablethroughoutArangoDBandFoxx.

Nodecompatibilitymodules

ArangoDBsupportsanumberofmodulesforcompatibilitywithNode.js,including:

assertimplementsbasicassertionandtestingfunctions.

bufferimplementsabinarydatatypeforJavaScript.

consoleisawellknownloggingfacilitytoalltheJavaScriptdevelopers.ArangoDBimplementsmostoftheConsoleAPI,withtheexceptionsofprofileandcount.

eventsimplementsaneventemitter.

fsprovidesafilesystemAPIforthemanipulationofpaths,directories,files,links,andtheconstructionoffilestreams.ArangoDBimplementsmostFilesystem/Afunctions.

moduleprovidesdirectaccesstothemodulesystem.

pathimplementsfunctionsdealingwithfilenamesandpaths.

punycodeimplementsconversionfunctionsforpunycodeencoding.

querystringprovidesutilitiesfordealingwithquerystrings.

streamprovidesastreaminginterface.

string_decoderimplementslogicfordecodingbuffersintostrings.

urlprovidesutilitiesforURLresolutionandparsing.

utilprovidesgeneralutilityfunctionslikeformatandinspect.

AdditionallyArangoDBprovidespartialimplementationsforthefollowingmodules:

net:onlyisIP,isIPv4andisIPv6.

process:onlyenvandcwd;stubsforargv,stdout.isTTY,stdout.write,nextTick.

timers:stubsforsetImmediate,setTimeout,setInterval,clearImmediate,clearTimeout,clearIntervalandref.

tty:onlyisatty(alwaysreturnsfalse).

vm:onlyrunInThisContext.

ThefollowingNode.jsmodulesarenotavailableatall:child_process,cluster,constants,crypto(butsee@arangodb/cryptobelow),dgram,dns,domain,http(butsee@arangodb/requestbelow),https,os,sys,tls,v8,zlib.

ArangoDBSpecificModulesTherearealargenumberofArangoDB-specificmodulesusingthe@arangodbnamespace,mostlyforinternalusebyArangoDBitself.Thefollowinghoweverarenoteworthy:

@arangodbprovidesdirectaccesstothedatabaseanditscollections.

@arangodb/cryptoprovidesvariouscryptographyfunctionsincludinghashingalgorithms.

JavaScriptModules

643

http://nodejs.org/api/assert.html

http://nodejs.org/api/buffer.html

http://wiki.commonjs.org/wiki/Console

http://nodejs.org/api/events.html

http://wiki.commonjs.org/wiki/Filesystem/A

http://nodejs.org/api/modules.html

http://nodejs.org/api/path.html

http://nodejs.org/api/punycode.html

http://en.wikipedia.org/wiki/Punycode

http://nodejs.org/api/querystring.html

http://nodejs.org/api/stream.html

https://nodejs.org/api/string_decoder.html

http://nodejs.org/api/url.html

http://nodejs.org/api/util.html

@arangodb/requestprovidesthefunctionalityformakingsynchronousHTTP/HTTPSrequests.

@arangodb/foxxisthenamespaceprovidingthevariousbuildingblocksoftheFoxxmicroserviceframework.

BundledNPMModules

ThefollowingNPMmodulesarepreinstalled:

aqbistheArangoDBQueryBuilderandcanbeusedtoconstructAQLquerieswithachainingJavaScriptAPI.

chaiisafull-featuredassertionlibraryforwritingJavaScripttests.

dedentisasimpleutilityfunctionforformattingmulti-linestrings.

error-stack-parserparsesstacktracesintoamoreusefulformat.

graphql-syncisanArangoDB-compatibleGraphQLserver/schemaimplementation.

highlight.jsisanHTMLsyntaxhighlighter.

i(inflect)isautilitylibraryforinflecting(e.g.pluralizing)words.

iconv-liteisautilitylibraryforconvertingbetweencharacterencodings

joiisavalidationlibrarythatissupportedthroughouttheFoxxframework.

js-yamlisaJavaScriptimplementationoftheYAMLdataformat(apartialsupersetofJSON).

lodashisautilitybeltforJavaScriptprovidingvarioususefulhelperfunctions.

minimatchisaglobmatcherformatchingwildcardsinfilepaths.

qsprovidesutilitiesfordealingwithquerystringsusingadifferentformatthanthequerystringmodule.

semverisautilitylibraryforhandlingsemverversionnumbers.

sinonisamockinglibraryforwritingteststubs,mocksandspies.

timezoneisalibraryforconvertingdatetimevaluesbetweenformatsandtimezones.

JavaScriptModules

644

https://www.npmjs.com

https://github.com/arangodb/aqbjs

http://chaijs.com

https://github.com/dmnd/dedent

http://www.stacktracejs.com

https://github.com/arangodb/graphql-sync

https://highlightjs.org

https://github.com/pksunkara/inflect

https://github.com/ashtuchkin/iconv-lite

https://github.com/hapijs/joi

https://github.com/nodeca/js-yaml

https://lodash.com

https://github.com/isaacs/minimatch

https://github.com/hapijs/qs

https://github.com/npm/node-semver

http://sinonjs.org

https://github.com/bigeasy/timezone

ArangoDBModuleconstarangodb=require('@arangodb')

Note:ThismoduleshouldnotbeconfusedwiththearangojsJavaScriptdriverwhichcanbeusedtoaccessArangoDBfromoutsidethedatabase.AlthoughtheAPIssharesimilaritiesandthefunctionalityoverlaps,thetwoarenotcompatiblewitheachotherandcannotbeusedinterchangeably.

The dbobjectarangodb.db

Thedbobjectrepresentsthecurrentdatabaseandletsyouaccesscollectionsandrunqueries.Formoreinformationseethedbobjectreference.

Examples


constthirteen=db._query('RETURN5+8').next();

The aqltemplatestringhandlerarangodb.aql

TheaqlfunctionisaJavaScripttemplatestringhandler.ItcanbeusedtowritecomplexAQLqueriesasmulti-linestringswithouthavingtoworryaboutbindVarsandthedistinctionbetweencollectionsandregularparameters.

TouseitjustprefixaJavaScripttemplatestring(theoneswithbackticksinsteadofquotes)withitsimportname(e.g.aql)andpassinvariableslikeyouwouldwitharegulartemplatestring.ThestringwillautomaticallybeconvertedintoanobjectwithqueryandbindVarsattributeswhichyoucanpassdirectlytodb._querytoexecute.Ifyoupassinacollectionitwillbeautomaticallyrecognizedasacollectionreferenceandhandledaccordingly.

Youcanalsousetheaql.literalhelpertomarkstringscontainingAQLsnippetsthatshouldbeinlineddirectlyintothequeryratherthanbetreatedasbindvariables.

aql.literalallowsyoutopassaarbitrarystringsintoyourAQLandthuswillopenyoutoAQLinjectionattacksifyouarepassinginuntrusteduserinputunsanitized.

TofindoutmoreaboutAQLseetheAQLdocumentation.

Examples


constfilterValue=23;

constmydata=db._collection('mydata');

constresult=db._query(aql`

FORdIN${mydata}

FILTERd.num>${filterValue}

RETURNd

`).toArray();

constfilterGreen=aql.literal('FILTERd.color=="green"');

constresult2=db._query(aql`

FORdIN${mydata}

${filterGreen}

RETURNd

`).toArray()

@arangodb

645

https://github.com/arangodb/arangojs

The queryhelperarangodb.query

Inmostcasesyouwilllikelyusetheaqltemplatehandlertocreateaqueryyoudirectlypasstodb._query.TomakethiseveneasierArangoDBprovidesthequerytemplatehandler,whichbehavesexactlylikeaqlbutalsodirectlyexecutesthequeryandreturnstheresultcursorinsteadofthequeryobject:

constquery=require('@arangodb').query;

constfilterValue=23;

constmydata=db._collection('mydata');

constresult=query`

FORdIN${mydata}

FILTERd.num>${filterValue}

RETURNd

`.toArray();

The errorsobjectarangodb.errors

ThisobjectprovidesusefulobjectsforeacherrorcodeArangoDBmightuseinArangoErrorerrors.ThisishelpfulwhentryingtocatchspecificerrorsraisedbyArangoDB,e.g.whentryingtoaccessadocumentthatdoesnotexist.EachobjecthasacodepropertycorrespondingtotheerrorNumfoundonArangoErrorerrors.

Foracompletelistoftheerrornamesandcodesyoumayencounterseetheappendixonerrorcodes.

Examples

consterrors=require('@arangodb').errors;

try{

someCollection.document('does-not-exist');

}catch(e){

if(e.isArangoError&&e.errorNum===errors.ERROR_ARANGO_DOCUMENT_NOT_FOUND.code){

thrownewError('Documentdoesnotexist');

}

thrownewError('Somethingwentwrong');

}

The timefunctionarangodb.time

Thisfunctionprovidesthecurrenttimeinsecondsasafloatingpointvaluewithmicrosecondprecisison.

ThisfunctioncanbeusedinsteadofDate.now()whenadditionalprecisionisneeded.

Examples

consttime=require('@arangodb').time;

conststart=time();

db._query(someVerySlowQuery);

console.log(`Elapsedtime:${time()-start}secs`);

@arangodb

646

ConsoleModuleglobal.console===require('console')

Note:Youdon'tneedtoloadthismoduledirectly.TheconsoleobjectisgloballydefinedthroughoutArangoDBandprovidesaccesstoallfunctionsinthismodule.

console.assert

console.assert(expression,format,argument1,...)

Teststhatanexpressionistrue.Ifnot,logsamessageandthrowsanexception.

Examples

console.assert(value==="abc","expected:value===abc,actual:",value);

console.debug

console.debug(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultasdebugmessage.Notethatdebugmessageswillonlybeloggediftheserverisstartedwithloglevelsdebugortrace.

Stringsubstitutionpatterns,whichcanbeusedinformat.

%%sstring%%d,%%iinteger%%ffloatingpointnumber%%oobjecthyperlink

Examples

console.debug("%s","thisisatest");

console.dir

console.dir(object)

Logsalistingofallpropertiesoftheobject.

Exampleusage:

console.dir(myObject);

console.errorconsole.error(format,argument1,...)

Formatstheargumentsaccordingto@FA{format}andlogstheresultaserrormessage.



Exampleusage:

console.error("error'%s':%s",type,message);

console

647

console.getline

console.getline()

Readsinalinefromtheconsoleandreturnsitasstring.

console.groupconsole.group(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultaslogmessage.Opensanestedblocktoindentallfuturemessagessent.CallgroupEndtoclosetheblock.Representationofblockisuptotheplatform,itcanbeaninteractiveblockorjustasetofindentedsubmessages.

Exampleusage:

console.group("userattributes");

console.log("name",user.name);

console.log("id",user.id);

console.groupEnd();

console.groupCollapsedconsole.groupCollapsed(format,argument1,...)

Sameasconsole.group.

console.groupEnd

console.groupEnd()

Closesthemostrecentlyopenedblockcreatedbyacalltogroup.

console.infoconsole.info(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultasinfomessage.



Exampleusage:

console.info("The%sjumpedover%dfences",animal,count);

console.log

console.log(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultaslogmessage.Thisisanaliasforconsole.info.

console.timeconsole.time(name)

Createsanewtimerunderthegivenname.CalltimeEndwiththesamenametostopthetimerandlogthetimeelapsed.

Exampleusage:

console.time("mytimer");

...

console

648

console.timeEnd("mytimer");//thiswillprinttheelapsedtime

console.timeEndconsole.timeEnd(name)

Stopsatimercreatedbyacalltotimeandlogsthetimeelapsed.

console.timeEnd

console.trace()

LogsastacktraceofJavaScriptexecutionatthepointwhereitiscalled.

console.warnconsole.warn(format,argument1,...)

Formatstheargumentsaccordingtoformatandlogstheresultaswarnmessage.



console

649

CryptoModuleconstcrypto=require('@arangodb/crypto')

Thecryptomoduleprovidesimplementationsofvarioushashingalgorithmsaswellascryptographyrelatedfunctions.

Nonces

Thesefunctionsdealwithcryptographicnonces.

createNonce

crypto.createNonce():string

Createsacryptographicnonce.

Returnsthecreatednonce.

checkAndMarkNonce

crypto.checkAndMarkNonce(nonce):void

Checksandmarksanonce.

Arguments

nonce:string

Thenoncetocheckandmark.

Returnsnothing.

Randomvalues

Thefollowingfunctionsdealwithgeneratingrandomvalues.

rand

crypto.rand():number

Generatesarandomintegerthatmaybepositive,negativeorevenzero.

Returnsthegeneratednumber.

genRandomAlphaNumbers

crypto.genRandomAlphaNumbers(length):string

Generatesastringofrandomalpabeticalcharactersanddigits.

Arguments

length:number

Thelengthofthestringtogenerate.

Returnsthegeneratedstring.

genRandomNumbers

crypto.genRandomNumbers(length):string

Generatesastringofrandomdigits.

crypto

650

Arguments

length:number



genRandomSalt

crypto.genRandomSalt(length):string

Generatesastringofrandom(printable)ASCIIcharacters.

Arguments

length:number



JSONWebTokens(JWT)

ThesemethodsimplementtheJSONWebTokenstandard.

jwtEncode

crypto.jwtEncode(key,message,algorithm):string

GeneratesaJSONWebTokenforthegivenmessage.

Arguments

key:string|null

Thesecretcryptographickeytobeusedtosignthemessageusingthegivenalgorithm.Notethatthisfunctionwillraiseanerrorifthekeyisomittedbutthealgorithmexpectsakey,andalsoifthealgorithmdoesnotexpectakeybutakeyisprovided(e.g.whenusing"none").

message:string

MessagetobeencodedasJWT.Notethatthemessagewillonlybebase64-encodedandsigned,notencrypted.Donotstoresensitiveinformationintokensunlesstheywillonlybehandledbytrustedparties.

algorithm:string

Nameofthealgorithmtouseforsigningthemessage,e.g."HS512".

ReturnstheJSONWebToken.

jwtDecode

crypto.jwtDecode(key,token,noVerify):string|null

Arguments

key:string|null

Thesecretcryptographickeythatwasusedtosignthemessageusingthealgorithmindicatedbythetoken.Notethatthisfunctionwillraiseanerrorifthekeyisomittedbutthealgorithmexpectsakey.

Ifthealgorithmdoesnotexpectakeybutakeyisprovided,thetokenwillfailtoverify.

token:string

Thetokentodecode.

crypto

651

Notethatthefunctionwillraiseanerrorifthetokenismalformed(e.g.doesnothaveexactlythreesegments).

noVerify:boolean(Default:false)

Whetherverificationshouldbeskipped.Ifthisissettotruethesignatureofthetokenwillnotbeverified.Otherwisethefunctionwillraiseanerrorifthesignaturecannotbeverifiedusingthegivenkey.

ReturnsthedecodedJSONmessageornullifnotokenisprovided.

jwtAlgorithms

AhelperobjectcontainingthesupportedJWTalgorithms.EachattributenamecorrespondstoaJWTalgandthevalueisanobjectwithsignandverifymethods.

jwtCanonicalAlgorithmName

crypto.jwtCanonicalAlgorithmName(name):string

AhelperfunctionthattranslatesaJWTalgvaluefoundinaJWTheaderintothecanonicalnameofthealgorithminjwtAlgorithms.Raisesanerrorifnoalgorithmwithamatchingnameisfound.

Arguments

name:string

Algorithmnametolookup.

Returnsthecanonicalnameforthealgorithm.

Hashingalgorithms

md5crypto.md5(message):string

HashesthegivenmessageusingtheMD5algorithm.

Arguments

message:string

Themessagetohash.

Returnsthecryptographichash.

sha1

crypto.sha1(message):string

HashesthegivenmessageusingtheSHA-1algorithm.

Arguments

message:string

Themessagetohash.


sha224



Arguments

crypto

652

message:string

Themessagetohash.


sha256crypto.sha256(message):string


Arguments

message:string

Themessagetohash.


sha384crypto.sha384(message):string


Arguments

message:string

Themessagetohash.


sha512



Arguments

message:string

Themessagetohash.


Miscellaneous

constantEqualscrypto.constantEquals(str1,str2):boolean

Comparestwostrings.Thisfunctioniteratesovertheentirelengthofbothstringsandcanhelpmakingcertaintimingattacksharder.

Arguments

str1:string

Thefirststringtocompare.

str2:string

Thesecondstringtocompare.

Returnstrueifthestringsareequal,falseotherwise.

pbkdf2

crypto

653

crypto.pbkdf2(salt,password,iterations,keyLength):string

GeneratesaPBKDF2-HMAC-SHA1hashofthegivenpassword.

Arguments

salt:string

Thecryptographicsalttohashthepasswordwith.

password:string

Themessageorpasswordtohash.

iterations:number

Thenumberofiterations.Thisshouldbeaveryhighnumber.OWASPrecommended64000iterationsin2012andrecommendsdoublingthatnumbereverytwoyears.

WhenusingPBKDF2forpasswordhashesitisalsorecommendedtoaddarandomvalue(typicallybetween0and32000)tothatnumberthatisdifferentforeachuser.

keyLength:number

Thekeylength.


hmac

crypto.hmac(key,message,algorithm):string

GeneratesanHMAChashofthegivenmessage.

Arguments

key:string

Thecryptographickeytousetohashthemessage.

message:string

Themessagetohash.

algorithm:string

Thenameofthealgorithmtouse.


crypto

654

FilesystemModuleconstfs=require('fs')

TheimplementationtriestofollowtheCommonJSFilesystem/A/0specificationwherepossible.

WorkingDirectory

Thedirectoryfunctionsbelowshouldn'tusethecurrentworkingdirectoryoftheserverlike.or./test.Youwillnotbeabletotellwhethertheenvironmenttheserverisrunninginwillpermitdirectorylisting,readingorwritingoffiles.

YoushouldeitherbaseyourdirectorieswithgetTempPath(),orasafoxxserviceusethemodule.context.basePath.

SingleFileDirectoryManipulation

exists

checksifafileofanytypeordirectoryexistsfs.exists(path)

Returnstrueifafile(ofanytype)oradirectoryexistsatagivenpath.Ifthefileisabrokensymboliclink,returnsfalse.

isFile

testsifpathisafilefs.isFile(path)

Returnstrueifthepathpointstoafile.

isDirectory

testsifpathisadirectoryfs.isDirectory(path)

Returnstrueifthepathpointstoadirectory.

size

getsthesizeofafilefs.size(path)

Returnsthesizeofthefilespecifiedbypath.

mtime

getsthelastmodificationtimeofafilefs.mtime(filename)

Returnsthelastmodificationdateofthespecifiedfile.ThedateisreturnedasaUnixtimestamp(numberofsecondselapsedsinceJanuary11970).

pathSeparatorfs.pathSeparator

Ifyouwanttocombinetwopathsyoucanusefs.pathSeparatorinsteadof/or\.

join

fs.join(path,filename)

Thefunctionreturnsthecombinationofthepathandfilename,e.g.fs.join(Hello/World,foo.bar)wouldreturnHello/World/foo.bar.

getTempFile

fs

655

http://wiki.commonjs.org/wiki/Filesystem/A/0

returnsthenamefora(new)temporaryfilefs.getTempFile(directory,createFile)

Returnsthenameforanewtemporaryfileindirectorydirectory.IfcreateFileistrue,anemptyfilewillbecreatedsonootherprocesscancreateafileofthesamename.

Note:Thedirectorydirectorymustexist.

getTempPath

returnsthetemporarydirectoryfs.getTempPath()

Returnstheabsolutepathofthetemporarydirectory

makeAbsolute

makesagivenpathabsolutefs.makeAbsolute(path)

Returnsthegivenstringifitisanabsolutepath,otherwiseanabsolutepathtothesamelocationisreturned.

chmod

setsfilepermissionsofspecifiedfiles(nonwindowsonly)fs.exists(path)

Returnstrueonsuccess.

list

returnsthedirectorylistingfs.list(path)

Thefunctionsreturnsthenamesofallthefilesinadirectory,inlexicallysortedorder.Throwsanexceptionifthedirectorycannotbetraversed(orpathisnotadirectory).

Note:thismeansthatlist("x")ofadirectorycontaining"a"and"b"wouldreturn["a","b"],not["x/a","x/b"].

listTree

returnsthedirectorytreefs.listTree(path)

Thefunctionreturnsanarraythatstartswiththegivenpath,andallofthepathsrelativetothegivenpath,discoveredbyadepthfirsttraversalofeverydirectoryinanyvisiteddirectory,reportingbutnottraversingsymboliclinkstodirectories.Thefirstpathisalways"" ,thepathrelativetoitself.

makeDirectory

createsadirectoryfs.makeDirectory(path)

Createsthedirectoryspecifiedbypath.

makeDirectoryRecursive

createsadirectoryfs.makeDirectoryRecursive(path)

Createsthedirectoryhierarchyspecifiedbypath.

remove

removesafilefs.remove(filename)

Removesthefilefilenameatthegivenpath.Throwsanexceptionifthepathcorrespondstoanythingthatisnotafileorasymboliclink.If"path"referstoasymboliclink,removesthesymboliclink.

removeDirectory

fs

656

removesanemptydirectoryfs.removeDirectory(path)

Removesadirectoryifitisempty.Throwsanexceptionifthepathisnotanemptydirectory.

removeDirectoryRecursive

removesadirectoryfs.removeDirectoryRecursive(path)

Removesadirectorywithallsubelements.Throwsanexceptionifthepathisnotadirectory.

FileIO

read

readsinafilefs.read(filename)

Readsinafileandreturnsthecontentasstring.PleasenotethatthefilecontentmustbeencodedinUTF-8.

read64

readsinafileasbase64fs.read64(filename)

Readsinafileandreturnsthecontentasstring.ThefilecontentisBase64encoded.

readBuffer

readsinafilefs.readBuffer(filename)

ReadsinafileandreturnsitscontentinaBufferobject.

readFileSyncfs.readFileSync(filename,encoding)

Readsthecontentsofthefilespecifiedinfilename.Ifencodingisspecified,thefilecontentswillbereturnedasastring.Supportedencodingsare:

utf8orutf-8ascii

base64

ucs2orucs-2utf16leorutf16behex

Ifnoencodingisspecified,thefilecontentswillbereturnedinaBufferobject.

write

fs.write(filename,content)

Writesthecontentintoafile.ContentcanbeastringoraBufferobject.Ifthefilealreadyexists,itistruncated.

writeFileSync

fs.writeFileSync(filename,content)

Thisisanaliasforfs.write(filename,content).

appendfs.append(filename,content)

Writesthecontentintoafile.ContentcanbeastringoraBufferobject.Ifthefilealreadyexists,thecontentisappendedattheend.

fs

657

RecursiveManipulation

copyRecursive

copiesadirectorystructurefs.copyRecursive(source,destination)

Copiessourcetodestination.Exceptionswillbethrownon:

Failuretocopythefilespecifyingadirectoryfordestinationwhensourceisafilespecifyingadirectoryassourceanddestination

CopyFile

copiesafileintoatargetfilefs.copyFile(source,destination)

Copiessourcetodestination.IfDestinationisadirectory,afileofthesamenamewillbecreatedinthatdirectory,elsethecopywillgetthespecifiedfilename.

move

renamesafilefs.move(source,destination)

Movessourcetodestination.Failuretomovethefile,orspecifyingadirectoryfordestinationwhensourceisafilewillthrowanexception.Likewise,specifyingadirectoryassourceanddestinationwillfail.

ZIP

unzipFile

unzipsafilefs.unzipFile(filename,outpath,skipPaths,overwrite,password)

Unzipsthezipfilespecifiedbyfilenameintothepathspecifiedbyoutpath.Overwritesanyexistingtargetfilesifoverwriteissettotrue.

Returnstrueifthefilewasunzippedsuccessfully.

zipFile

zipsafilefs.zipFile(filename,chdir,files,password)

Storesthefilesspecifiedbyfilesinthezipfilefilename.Ifthefilefilenamealreadyexists,anerroristhrown.Thelistofinputfilesfilesmustbegivenasalistofabsolutefilenames.Ifchdirisnotempty,thechdirprefixwillbestrippedfromthefilenameinthezipfile,sowhenitisunzippedfilenameswillberelative.Specifyingapasswordisoptional.

Returnstrueifthefilewaszippedsuccessfully.

fs

658

RequestModuleconstrequest=require('@arangodb/request')

TherequestmoduleprovidesthefunctionalityformakingHTTPrequests.

MakingHTTPrequests

HTTPmethodhelpers

InadditiontotherequestfunctionconvenienceshorthandsareavailableforeachHTTPmethodintheformof,i.e.:

request.head(url,options)

request.get(url,options)

request.post(url,options)

request.put(url,options)

request.delete(url,options)

request.patch(url,options)

Theseareequivalenttousingtherequestfunctiondirectly,i.e.:

request[method](url,options)

//isequivalentto

request({method,url,...options});

Forexample:


request.get('http://localhost',{headers:{'x-session-id':'keyboardcat'}});

//isequivalentto

request({

method:'get',

url:'http://localhost',

headers:{'x-session-id':'keyboardcat'}

});

Therequestfunction

TherequestfunctioncanbeusedtomakeHTTPrequests.

request(options)

PerformsanHTTPrequestandreturnsaResponseobject.

Parameter

Therequestfunctiontakesthefollowingoptions:

urloruri:thefully-qualifiedURLoraparsedURLfromurl.parse.qs(optional):objectcontainingquerystringvaluestobeappendedtotheURL.useQuerystring:iftrue,usequerystringmoduletohandlequerystrings,otherwiseuseqsmodule.Default:false.method(optional):HTTPmethod(case-insensitive).Default:"GET".headers(optional):HTTPheaders(case-insensitive).Default:{}.body(optional):requestbody.MustbeastringorBuffer,oraJSONserializablevalueifjsonistrue.json:iftrue,bodywillbeserializedtoaJSONstringandtheContent-Typeheaderwillbesetto"application/json".AdditionallytheresponsebodywillalsobeparsedasJSON(unlessencodingissettonull).Default:false.form(optional):whensettoastringorobjectandnobodyhasbeenset,bodywillbesettoaquerystringrepresentationofthatvalueandtheContent-Typeheaderwillbesetto"application/x-www-form-urlencoded".AlsoseeuseQuerystring.auth(optional):anobjectwiththepropertiesusernameandpasswordforHTTPBasicauthenticationorthepropertybearerfor

request

659

HTTPBearertokenauthentication.

sslProtocol(optional):whichtlsversionshouldbeusedtoconnecttotheurl.Thedefaultis4whichisTLS1.0.Seesslprotocolformoreopitions.followRedirect:whetherHTTP3xxredirectsshouldbefollowed.Default:true.maxRedirects:themaximumnumberofredirectstofollow.Default:10.encoding:encodingtobeusedfortheresponsebody.Ifsettonull,theresponsebodywillbereturnedasaBuffer.Default:"utf-8".timeout:numberofmillisecondstowaitforaresponsebeforeabortingtherequest.returnBodyOnError:whethertheresponsebodyshouldbereturnedevenwhentheserverresponseindicatesanerror.Default:true.

ThefunctionreturnsaResponseobjectwiththefollowingproperties:

rawBody:therawresponsebodyasaBuffer.body:theparsedresponsebody.Ifencodingwasnotsettonull,thisisastring.Ifadditionallyjsonwassettotrueandtheresponsebodyiswell-formedJSON,thisistheparsedJSONdata.headers:anobjectcontainingtheresponseheaders.OtherwisethisisidenticaltorawBody.statusCodeandstatus:theHTTPstatuscodeoftheresponse,e.g.404.message:theHTTPstatusmessageoftheresponse,e.g.NotFound.

Forms

Therequestmodulesupportsapplication/x-www-form-urlencoded(URLencoded)formuploads:


varres=request.post('http://service.example/upload',{form:{key:'value'}});

//or

varres=request.post({url:'http://service.example/upload',form:{key:'value'}});

//or

varres=request({

method:'post',

url:'http://service.example/upload',

form:{key:'value'}

});

Formdatawillbeencodedusingtheqsmodulebydefault.

Ifyouwanttousethequerystringmoduleinstead,simplyusetheuseQuerystringoption.

JSON

IfyouwanttosubmitJSON-serializablevaluesasrequestbodies,justsetthejsonoption:


varres=request.post('http://service.example/notify',{body:{key:'value'},json:true});

//or

varres=request.post({url:'http://service.example/notify',body:{key:'value'},json:true});

//or

varres=request({

method:'post',

url:'http://service.example/notify',

body:{key:'value'},

json:true

});

HTTPauthentication

TherequestmodulesupportsbothHTTPBasicauthentication.Justpassthecredentialsviatheauthoption:


varres=request.get(

request

660

https://www.npmjs.com/package/qs

http://nodejs.org/api/querystring.html

'http://service.example/secret',

{auth:{username:'jcd',password:'bionicman'}}

);

//or

varres=request.get({

url:'http://service.example/secret',

auth:{username:'jcd',password:'bionicman'}

});

//or

varres=request({

method:'get',


auth:{username:'jcd',password:'bionicman'}

});

AlternativelyyoucansupplythecredentialsviatheURL:


varusername='jcd';

varpassword='bionicman';

varres=request.get(

'http://'+

encodeURIComponent(username)+

':'+

encodeURIComponent(password)+

'@service.example/secret'

);

YoucanalsouseBearertokenauthentication:


varres=request.get(

'http://service.example/secret',

{auth:{bearer:'keyboardcat'}}

);

//or

varres=request.get({


auth:{bearer:'keyboardcat'}

});

//or

varres=request({

method:'get',


auth:{bearer:'keyboardcat'}

});

request

661

Module"actions"constactions=require('@arangodb/actions')

Theactionmoduleprovidestheinfrastructurefordefininglow-levelHTTPactions.

IfyouwanttodefineHTTPendpointsinArangoDByoushouldprobablyusetheFoxxmicroserviceframeworkinstead.

Basics

Errormessage

actions.getErrorMessage(code)

Returnstheerrormessageforanerrorcode.

StandardHTTPResultGeneratorsactions.defineHttp(options)

Definesanewaction.Theoptionsareasfollows:

options.url

TheURL,whichcanbeusedtoaccesstheaction.Thispathmightcontainslashes.Notethatthisactionwillalsobecalled,ifaurlisgivensuchthatoptions.urlisaprefixofthegivenurlandnolongerdefinitionmatches.

options.prefix

Iffalse,thenonlyusetheactionforexactmatches.Thedefaultistrue.

options.callback(request,response)

Therequestargumentcontainsadescriptionoftherequest.Arequestparameterfooisaccessibleasrequest.parametrs.foo.Arequestheaderbarisaccessibleasrequest.headers.bar.Assumethattheactionisdefinedfortheurl/foo/barandtherequesturlis/foo/bar/hugo/egon.Thenthesuffixparts["hugo","egon"]areavailibleinrequest.suffix.

Thecallbackmustdefinefilltheresponse.

response.responseCode:theresponsecoderesponse.contentType:thecontenttypeoftheresponseresponse.body:thebodyoftheresponse

YoucanusethefunctionsResultOkandResultErrortoeasilygeneratearesponse.

Resultok

actions.resultOk(req,res,code,result,headers)

Thefunctiondefinesaresponse.codeisthestatuscodetoreturn.resultistheresultobject,whichwillbereturnedasJSONobjectinthebody.headersisanarrayofheaderstoreturned.Thefunctionaddstheattributeerrorwithvaluefalseandcodewithvaluecodetotheresult.

Resultbad

actions.resultBad(req,res,error-code,msg,headers)

Thefunctiongeneratesanerrorresponse.

Resultnotfoundactions.resultNotFound(req,res,code,msg,headers)

actions

662


Resultunsupported

actions.resultUnsupported(req,res,headers)


Resulterror

actions.resultError(req,res,code,errorNum,errorMessage,headers,keyvals)*

Thefunctiongeneratesanerrorresponse.TheresponsebodyisanarraywithanattributeerrorMessagecontainingtheerrormessageerrorMessage,errorcontainingtrue,codecontainingcode,errorNumcontainingerrorNum,anderrorMessagecontainingtheerrormessageerrorMessage.keyvalsaremixedintotheresult.

ResultnotImplementedactions.resultNotImplemented(req,res,msg,headers)


Resultpermanentredirect

actions.resultPermanentRedirect(req,res,options,headers)

Thefunctiongeneratesaredirectresponse.

Resulttemporaryredirectactions.resultTemporaryRedirect(req,res,options,headers)

Thefunctiongeneratesaredirectresponse.

ArangoDBResultGenerators

Collectionnotfoundactions.collectionNotFound(req,res,collection,headers)


Indexnotfound

actions.indexNotFound(req,res,collection,index,headers)


Resultexceptionactions.resultException(req,res,err,headers,verbose)

Thefunctiongeneratesanerrorresponse.If@FA{verbose}issettotrueornotspecified(thedefault),thentheerrorstacktracewillbeincludedintheerrormessageifavailable.If@FA{verbose}isastringitwillbeprependedbeforetheerrormessageandthestacktracewillalsobeincluded.

actions

663

QueriesModuleconstqueries=require('@arangodb/aql/queries')

ThequerymoduleprovidestheinfrastructureforworkingwithcurrentlyrunningAQLqueriesviaarangosh.

Properties

queries.properties()Returnstheserverscurrentquerytrackingconfiguration;wechangetheslowquerythresholdtogetbetterresults:

arangosh>varqueries=require("@arangodb/aql/queries");

arangosh>queries.properties();

arangosh>queries.properties({slowQueryThreshold:1});


Currentlyrunningqueries

Wecreateataskthatspawnsqueries,sowehaveniceoutput.Sincethistaskusesresources,youmaywanttoincreaseperiod(andnotforgettoremoveit...afterwards):

arangosh>vartheQuery='FORsleepLoooongIN1..5LETsleepLoooonger=SLEEP(1000)RETURN

sleepLoooong';

arangosh>vartasks=require("@arangodb/tasks");

arangosh>tasks.register({

........>id:"mytask-1",

........>name:"thisisasampletasktospawnaslowaqlquery",

........>command:"require('@arangodb').db._query('"+theQuery+"');"

........>});

arangosh>queries.current();

showexecutionresultsThefunctionreturnsthecurrentlyrunningAQLqueriesasanarray.

SlowqueriesThefunctionreturnsthelastAQLqueriesthatexceededtheslowquerythresholdasanarray:

arangosh>queries.slow();

[]

ClearslowqueriesClearthelistofslowAQLqueries:

arangosh>queries.clearSlow();

{

"code":200

}

arangosh>queries.slow();

[]

queries

664

KillKillarunningAQLquery:

arangosh>varrunningQueries=queries.current().filter(function(query){

........>returnquery.query===theQuery;

........>});

arangosh>queries.kill(runningQueries[0].id);

{

"code":200

}

queries

665

Write-aheadlogconstwal=require('internal').wal

Thismoduleprovidesfunctionalityforadministeringthewrite-aheadlogs.Mostofthesefunctionsonlyreturnsensiblevalueswheninvokedwiththemmfilesenginebeingactive.

Configuration

retrievestheconfigurationofthewrite-aheadloginternal.wal.properties()

Retrievestheconfigurationofthewrite-aheadlog.TheresultisaJSONarraywiththefollowingattributes:

allowOversizeEntries:whetherornotoperationsthatarebiggerthanasinglelogfilecanbeexecutedandstoredlogfileSize:thesizeofeachwrite-aheadlogfilehistoricLogfiles:themaximumnumberofhistoriclogfilestokeepreserveLogfiles:themaximumnumberofreservelogfilesthatArangoDBallocatesinthebackgroundsyncInterval:theintervalforautomaticsynchronizationofnot-yetsynchronizedwrite-aheadlogdata(inmilliseconds)throttleWait:themaximumwaittimethatoperationswillwaitbeforetheygetabortedifcaseofwrite-throttling(inmilliseconds)throttleWhenPending:thenumberofunprocessedgarbage-collectionoperationsthat,whenreached,willactivatewrite-throttling.Avalueof0meansthatwrite-throttlingwillnotbetriggered.

Examples

arangosh>require("internal").wal.properties();

showexecutionresultsconfiguresthewrite-aheadloginternal.wal.properties(properties)

Configuresthebehaviorofthewrite-aheadlog.propertiesmustbeaJSONJSONobjectwiththefollowingattributes:

allowOversizeEntries:whetherornotoperationsthatarebiggerthanasinglelogfilecanbeexecutedandstoredlogfileSize:thesizeofeachwrite-aheadlogfilehistoricLogfiles:themaximumnumberofhistoriclogfilestokeepreserveLogfiles:themaximumnumberofreservelogfilesthatArangoDBallocatesinthebackgroundthrottleWait:themaximumwaittimethatoperationswillwaitbeforetheygetabortedifcaseofwrite-throttling(inmilliseconds)throttleWhenPending:thenumberofunprocessedgarbage-collectionoperationsthat,whenreached,willactivatewrite-throttling.Avalueof0meansthatwrite-throttlingwillnotbetriggered.

Specifyinganyoftheaboveattributesisoptional.Notspecifiedattributeswillbeignoredandtheconfigurationforthemwillnotbemodified.

Examples

arangosh>require("internal").wal.properties({

........>allowOverSizeEntries:true,

........>logfileSize:32*1024*1024});


Flushing

flushesthecurrentlyopenWALlogfileinternal.wal.flush(waitForSync,waitForCollector)

Flushesthewrite-aheadlog.Byflushingthecurrentlyactivewrite-aheadlogfile,thedatainitcanbetransferredtocollectionjournalsanddatafiles.Thisisusefultoensurethatalldataforacollectionispresentinthecollectionjournalsanddatafiles,forexample,whendumpingthedataofacollection.

Write-aheadlog

666

ThewaitForSyncoptiondetermineswhetherornottheoperationshouldblockuntilthenot-yetsynchronizeddatainthewrite-aheadlogwassynchronizedtodisk.

ThewaitForCollectoroperationcanbeusedtospecifythattheoperationshouldblockuntilthedataintheflushedloghasbeencollectedbythewrite-aheadloggarbagecollector.Notethatsettingthisoptiontotruemightblockforalongtimeiftherearelong-runningtransactionsandthewrite-aheadloggarbagecollectorcannotfinishgarbagecollection.

Examples

arangosh>require("internal").wal.flush();

Write-aheadlog

667

TaskManagementconsttasks=require('@arangodb/tasks')

Note:IfyouaretryingtoscheduletasksinFoxxyoushouldconsiderusingtheFoxxqueuesmoduleinstead,whichprovidesamorehigh-levelAPIthatalsopersiststasksacrossreboots.

IntroductiontoTaskManagementinArangoDB

ArangoDBcanexecuteuser-definedJavaScriptfunctionsasone-shotorperiodictasks.Thisfunctionalitycanbeusedtoimplementtimedorrecurringjobsinthedatabase.

TasksinArangoDBconsistofaJavaScriptsnippetorfunctionthatisexecutedwhenthetaskisscheduled.Ataskcanbeaone-shottask(meaningitisrunonceandnotrepeated)oraperiodictask(meaningthatitisre-scheduledaftereachexecution).Taskscanhaveoptionalparameters,whicharedefinedattasksetuptime.Theparametersspecifiedattasksetuptimewillbepassedasargumentstothetaskwheneveritgetsexecuted.PeriodicTaskshaveanexecutionfrequencythatneedstobespecifiedwhenthetaskissetup.One-shottaskshaveaconfigurabledelayafterwhichthey'llgetexecuted.

Taskswillbeexecutedontheservertheyhavebeensetupon.Taskswillnotbeshippedaroundinacluster.Ataskwillbeexecutedinthecontextofthedatabaseitwascreatedin.However,whendroppingadatabase,anytasksthatwerecreatedinthecontextofthisdatabasewillremainactive.Itisthereforesensibletofirstunregisterallactivetasksforadatabasebeforedroppingthedatabase.

TasksregisteredinArangoDBwillbeexecuteduntiltheservergetsshutdownorrestarted.Afterarestartoftheserver,anyuser-definedone-shotorperiodictaskswillbelost.

CommandsforWorkingwithTasksArangoDBprovidesthefollowingcommandsforworkingwithtasks.Allcommandscanbeaccessedviathetasksmodule,whichcanbeloadedlikethis:

require("@arangodb/tasks")

PleasenotethatthetasksmoduleisavailableinsidetheArangoDBserveronly.ItcannotbeusedfromtheArangoShellorArangoDB'swebinterface.

Registeratask

Toregisteratask,theJavaScriptsnippetorfunctionneedstobespecifiedinadditiontotheexecutionfrequency.Optionally,ataskcanhaveanidandaname.Ifnoidisspecified,itwillbeauto-assignedforanewtask.Thetaskidisalsothemeanstoaccessorunregisteratasklater.Tasknamesareinformationalonly.Theycanbeusedtomakeataskdistinguishablefromothertasksalsorunningontheserver.

Thefollowingserver-sidecommandsregisteratask.ThecommandtobeexecutedisaJavaScriptstringsnippetwhichprintsamessagetotheserver'slogfile:

consttasks=require("@arangodb/tasks");

tasks.register({

id:"mytask-1",

name:"thisisasnippettask",

period:15,

command:"require('console').log('hellofromsnippettask');"

});

Theabovehasregisterataskwithidmytask-1,whichwillbeexecutedevery15secondsontheserver.Thetaskwillwritealogmessagewheneveritisinvoked.

TaskscanalsobesetupusingaJavaScriptcallbackfunctionlikethis:

TaskManagement

668


tasks.register({

id:"mytask-2",

name:"thisisafunctiontask",

period:15,

command:function(){

require('console').log('hellofromfunctiontask');

}

});

Itisimportanttonotethatthecallbackfunctionislateboundandwillbeexecutedinadifferentcontextthaninthecreationcontext.Thecallbackfunctionmustthereforenotaccessanyvariablesdefinedoutsideofitsownscope.Thecallbackfunctioncanstilldefineanduseitsownvariables.

Topassparameterstoatask,theparamsattributecanbesetwhenregisteringatask.NotethattheparametersarelimitedtodatatypesusableinJSON(meaningnocallbackfunctionscanbepassedasparametersintoatask):


tasks.register({

id:"mytask-3",

name:"thisisaparametertask",

period:15,


vargreeting=params.greeting;

vardata=JSON.stringify(params.data);

require('console').log('%sfromparametertask:%s',greeting,data);

},

params:{greeting:"hi",data:"howareyou?"}

});

Registeringaone-shottaskworksthesameway,exceptthattheperiodattributemustbeomitted.Ifperiodisomitted,thenthetaskwillbeexecutedjustonce.Thetaskinvocationdelaycanoptionallybespecifiedwiththeoffsetattribute:


tasks.register({

id:"mytask-once",

name:"thisisaone-shottask",

offset:10,


require('console').log('youwillseemejustonce!');

}

});

Note:Whenspecifyinganoffsetvalueof0,ArangoDBwillinternallyaddaverysmallvaluetotheoffsetsowillbeslightlygreaterthanzero.

Unregisteratask

Afterataskhasbeenregistered,itcanbeunregisteredusingitsid:


tasks.unregister("mytask-1");

Notethatunregisteringanon-existingtaskwillthrowanexception.

ListalltasksTogetanoverviewofwhichtasksareregistered,thereisthegetmethod.Ifthegetmethodiscalledwithoutanyarguments,itwillreturnanarrayofalltasks:

TaskManagement

669


tasks.get();

Ifgetiscalledwithataskidargument,itwillreturninformationaboutthisparticulartask:


tasks.get("mytask-3");

Thecreatedattributeofataskrevealswhenataskwascreated.ItisreturnedasaUnixtimestamp.

TaskManagement

670

DeprecatedFeatureslistedinthissectionshouldnolongerbeused,becausetheyareconsideredobsoleteandmaygetremovedinafuturerelease.Theyarecurrentlykeptforbackwardcompatibility.Thereareusuallybetteralternativestoreplacetheoldfeatureswith:

SimpleQueries:Ideomaticinterfaceinarangoshtoperformtrivialqueries.TheyaresupersededbyAQLqueries,whichcanalsoberuninarangosh.AQLisalanguageonitsownandwaymorepowerfulthanSimpleQueriescouldeverbe.Infact,the(stillsupported)SimpleQueriesaretranslatedinternallytoAQL,thentheAQLqueryisoptimizedandrunagainstthedatabaseinrecentversions,becauseofbetterperformanceandreducedmaintenancecomplexity.

Actions:SnippetsofJavaScriptcodeontheserver-sideforminimalcustomendpoints.SincetheFoxxrevampin3.0,itbecamereallyeasytowriteFoxxMicroservices,whichallowyoutodefinecustomendpointsevenwithcomplexbusinesslogic.

Deprecated

671

SimpleQueries

ItisrecommendedtouseAQLinstead.

Simplequeriescanbeusedifthequeryconditionisstraightforward,i.e.,adocumentreference,alldocuments,aquery-by-example,orasimplegeoquery.Inasimplequeryyoucanspecifyexactlyonecollectionandonequerycriteria.InthefollowingsectionswedescribetheJavaScriptshellinterfaceforsimplequeries,whichyoucanusewithintheArangoDBshellandwithinactionsandtransactions.ForotherlanguagesseethecorrespondinglanguageAPIdocumentation.

YoucanfindalistofqueriesatCollectionMethods.

SimpleQueries

672

SequentialAccessandCursorsIfaqueryreturnsacursor,thenyoucanusehasNextandnexttoiterateovertheresultsetortoArraytoconvertittoanarray.

Ifthenumberofqueryresultsisexpectedtobebig,itispossibletolimittheamountofdocumentstransferredbetweentheserverandtheclienttoaspecificvalue.ThisvalueiscalledbatchSize.ThebatchSizecanoptionallybesetbeforeorwhenasimplequeryisexecuted.Iftheserverhasmoredocumentsthanshouldbereturnedinasinglebatch,theserverwillsetthehasMoreattributeintheresult.Itwillalsoreturntheidoftheserver-sidecursorintheidattributeintheresult.ThisidcanbeusedwiththecursorAPItofetchanyoutstandingresultsfromtheserveranddisposetheserver-sidecursorafterwards.

TheinitialbatchSizevaluecanbesetusingthesetBatchSizemethodthatisavailableforeachtypeofsimplequery,orwhenthesimplequeryisexecutedusingitsexecutemethod.IfnobatchSizevalueisspecified,theserverwillpickareasonabledefaultvalue.

HasNext

checksifthecursorisexhaustedcursor.hasNext()

ThehasNextoperatorreturnstrue,thenthecursorstillhasdocuments.Inthiscasethenextdocumentcanbeaccessedusingthenextoperator,whichwilladvancethecursor.

Examples

arangosh>vara=db.five.all();

arangosh>while(a.hasNext())print(a.next());


Next

returnsthenextresultdocumentcursor.next()

IfthehasNextoperatorreturnstrue,thentheunderlyingcursorofthesimplequerystillhasdocuments.Inthiscasethenextdocumentcanbeaccessedusingthenextoperator,whichwilladvancetheunderlyingcursor.Ifyouusenextonanexhaustedcursor,thenundefinedisreturned.

Examples

arangosh>db.five.all().next();


SetBatchsize

setsthebatchsizeforanyfollowingrequestscursor.setBatchSize(number)

Setsthebatchsizeforqueries.Thebatchsizedetermineshowmanyresultsareatmosttransferredfromtheservertotheclientinonechunk.

GetBatchsize

returnsthebatchsizecursor.getBatchSize()

Returnsthebatchsizeforqueries.Ifthereturnedvalueisundefined,theserverwilldetermineasensiblebatchsizeforanyfollowingrequests.

ExecuteQuery

executesaqueryquery.execute(batchSize)

SimpleQueries

673

Executesasimplequery.IftheoptionalbatchSizevalueisspecified,theserverwillreturnatmostbatchSizevaluesinoneroundtrip.ThebatchSizecannotbeadjustedafterthequeryisfirstexecuted.

Note:Thereisnoneedtoexplicitlycalltheexecutemethodifanothermeansoffetchingthequeryresultsischosen.Thefollowingtwoapproachesleadtothesameresult:

arangosh>result=db.users.all().toArray();

arangosh>q=db.users.all();q.execute();result=[];while(q.hasNext()){

result.push(q.next());}

showexecutionresultsThefollowingtwoalternativesbothuseabatchSizeandreturnthesameresult:

arangosh>q=db.users.all();q.setBatchSize(20);q.execute();while(q.hasNext()){

print(q.next());}

arangosh>q=db.users.all();q.execute(20);while(q.hasNext()){print(q.next());}


Dispose

disposestheresultcursor.dispose()

Ifyouarenolongerinterestedinanyfurtherresults,youshouldcalldisposeinordertofreeanyresourcesassociatedwiththecursor.Aftercallingdisposeyoucannolongeraccessthecursor.

Count

countsthenumberofdocumentscursor.count()

Thecountoperatorcountsthenumberofdocumentintheresultsetandreturnsthatnumber.Thecountoperatorignoresanylimitsandreturnsthetotalnumberofdocumentsfound.

Note:Notallsimplequeriessupportcounting.Inthiscasenullisreturned.

cursor.count(true)

Iftheresultsetwaslimitedbythelimitoperatorordocumentswereskipedusingtheskipoperator,thecountoperatorwithargumenttruewillusethenumberofelementsinthefinalresultset-afterapplyinglimitandskip.

Note:Notallsimplequeriessupportcounting.Inthiscasenullisreturned.

Examples

Ignoreanylimit:

arangosh>db.five.all().limit(2).count();

null

Countinganylimitorskip:

arangosh>db.five.all().limit(2).count(true);

2

SimpleQueries

674

Pagination

ItisrecommendedtouseAQLinstead,seetheLIMIToperation.

If,forexample,youdisplaytheresultofausersearch,thenyouareingeneralnotinterestedinthecompletedresultset,butonlythefirst10orsodocuments.Ormaybethenext10documentsforthesecondpage.Inthiscase,youcantheskipandlimitoperators.TheseoperatorsworklikeLIMITinMySQL.

skipusedtogetherwithlimitcanbeusedtoimplementpagination.Theskipoperatorskipsoverthefirstndocuments.So,inordertocreateresultpageswith10resultdocumentsperpage,youcanuseskip(n*10).limit(10)toaccessthe10documentsonthenthpage.Thisresultshouldbesorted,sothatthepaginationworksinapredicableway.

Limit

limitquery.limit(number)

Limitsaresulttothefirstnumberdocuments.Specifyingalimitof0willreturnnodocumentsatall.Ifyoudonotneedalimit,justdonotaddthelimitoperator.Thelimitmustbenon-negative.

Ingeneraltheinputtolimitshouldbesorted.Otherwiseitwillbeunclearwhichdocumentswillbeincludedintheresultset.

Examples


arangosh>db.five.all().limit(2).toArray();


Skip

skipquery.skip(number)

Skipsthefirstnumberdocuments.Ifnumberispositive,thenthisnumberofdocumentsareskippedbeforereturningthequeryresults.

Ingeneraltheinputtoskipshouldbesorted.Otherwiseitwillbeunclearwhichdocumentswillbeincludedintheresultset.

Note:usingnegativeskipvaluesisdeprecatedasofArangoDB2.6andwillnotbesupportedinfutureversionsofArangoDB.

Examples


arangosh>db.five.all().skip(3).toArray();


SimpleQueries

675

ModificationQueries

ItisrecommendedtouseAQLinstead,seeDataModificationQueries.

ArangoDBalsoallowsremoving,replacing,andupdatingdocumentsbasedonanexampledocument.Everydocumentinthecollectionwillbecomparedagainstthespecifiedexampledocumentandbedeleted/replaced/updatedifallattributesmatch.

Thesemethodshouldbeusedwithcautionastheyareintendedtoremoveormodifylotsofdocumentsinacollection.

Allmethodscanoptionallyberestrictedtoaspecificnumberofoperations.However,ifalimitisspecificbutislessthanthenumberofmatches,itwillbeundefinedwhichofthematchingdocumentswillgetremoved/modified.RemovebyExample,ReplacebyExampleandUpdatebyExamplearedescribedwithexamplesinthesubchapterCollectionMethods.

SimpleQueries

676

GeoQueries

ItisrecommendedtouseAQLinstead,seeGeofunctions.

TheArangoDBallowstoselectdocumentsbasedongeographiccoordinates.Inorderforthistowork,ageo-spatialindexmustbedefined.ThisindexwilluseaveryelaboratealgorithmtolookupneighborsthatisamagnitudefasterthanasimpleR*index.

Ingeneralageocoordinateisapairoflatitudeandlongitude,whichmustbothbespecifiedasnumbers.Ageoindexcanbecreatedoncoordinatesthatarestoredinasinglelistattributewithtwoelementslike[-10,+30](latitudefirst,followedbylongitude)oroncoordinatesstoredintwoseparateattributes.

Forexample,toindexthefollowingdocuments,anindexcanbecreatedonthepositionattributeofthedocuments:

db.test.save({position:[-10,30]});

db.test.save({position:[10,45.5]});

db.test.ensureIndex({type:"geo",fields:["position"]});

Ifcoordinatesarestoredintwodistinctattributes,theindexmustbecreatedonthetwoattributes:

db.test.save({latitude:-10,longitude:30});

db.test.save({latitude:10,longitude:45.5});

db.test.ensureIndex({type:"geo",fields:["latitude","longitude"]});

Inordertofindalldocumentswithinagivenradiusaroundacoordinateusethewithinoperator.Inordertofindalldocumentsnearagivendocumentusethenearoperator.

Itispossibletodefinemorethanonegeo-spatialindexpercollection.Inthiscaseyoumustgiveahintusingthegeooperatorwhichofindexesshouldbeusedinaquery.

Near

constructsanearqueryforacollectioncollection.near(latitude,longitude)

Thereturnedlistissortedaccordingtothedistance,withthenearestdocumenttothecoordinate(latitude,longitude)comingfirst.Ifthereareneardocumentsofequaldistance,documentsarechosenrandomlyfromthissetuntilthelimitisreached.Itispossibletochangethelimitusingthelimitoperator.

Inordertousethenearoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.

Note:neardoesnotsupportnegativeskips.//However,youcanstilluselimitfollowedtoskip.

collection.near(latitude,longitude).limit(limit)

Limitstheresulttolimitdocumentsinsteadofthedefault100.

Note:Unlikewithmultipleexplicitlimits,limitwillraisetheimplicitdefaultlimitimposedbywithin.

collection.near(latitude,longitude).distance()

Thiswilladdanattributedistancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.

collection.near(latitude,longitude).distance(name)



SimpleQueries

677

Note:thenearsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthenearoperatoristousetheAQLNEARfunctioninanAQLqueryasfollows:


RETURNdoc

Examples

Togetthenearesttwolocations:



........>for(varj=-180;j<=180;j+=10){


........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}


showexecutionresultsIfyouneedthedistanceaswell,thenyoucanusethedistanceoperator:



........>for(varj=-180;j<=180;j+=10){


........>name:"Name/"+i+"/"+j,

........>loc:[i,j]});

........>}}

arangosh>db.geo.near(0,0).distance().limit(2).toArray();


Within

constructsawithinqueryforacollectioncollection.within(latitude,longitude,radius)

Thiswillfindalldocumentswithinagivenradiusaroundthecoordinate(latitude,longitude).Thereturnedarrayissortedbydistance,beginningwiththenearestdocument.

Inordertousethewithinoperator,ageoindexmustbedefinedforthecollection.Thisindexalsodefineswhichattributeholdsthecoordinatesforthedocument.Ifyouhavemorethenonegeo-spatialindex,youcanusethegeooperatortoselectaparticularindex.

collection.within(latitude,longitude,radius).distance()

Thiswilladdanattribute_distancetoalldocumentsreturned,whichcontainsthedistancebetweenthegivenpointandthedocumentinmeters.

collection.within(latitude,longitude,radius).distance(name)



Note:thewithinsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforretrievingdocumentsfromacollectionusingthewithinoperatoristousetheAQLWITHINfunctioninanAQLqueryasfollows:


SimpleQueries

678

RETURNdoc

Examples

Tofindalldocumentswithinaradiusof2000kmuse:


........>for(varj=-180;j<=180;j+=10){

........>db.geo.save({name:"Name/"+i+"/"+j,loc:[i,j]});}}

arangosh>db.geo.within(0,0,2000*1000).distance().toArray();


Geo

constructsageoindexselectioncollection.geo(location-attribute)

Looksupageoindexdefinedonattributelocation_attribute.


Thisisusefulforcollectionswithmultipledefinedgeoindexes.

collection.geo(location_attribute,true)

Looksupageoindexonacompoundattributelocation_attribute.


collection.geo(latitude_attribute,longitude_attribute)

Looksupageoindexdefinedonthetwoattributeslatitude_attributeandlongitude-attribute.



Note:thegeosimplequeryhelperfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforrunninggeoqueriesistousetheirAQLequivalents.

Examples

Assumeyouhavealocationstoredaslistintheattributehomeandadestinationstoredintheattributework.Thenyoucanusethegeooperatortoselectwhichgeo-spatialattributes(andthuswhichindex)touseinanearquery.


........>for(j=-180;j<=180;j+=10){

........>db.complex.save({name:"Name/"+i+"/"+j,

........>home:[i,j],

........>work:[-i,-j]});

........>}

........>}

........>

arangosh>db.complex.near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["home"]});

arangosh>db.complex.near(0,170).limit(5).toArray();

arangosh>db.complex.geo("work").near(0,170).limit(5);

arangosh>db.complex.ensureIndex({type:"geo",fields:["work"]});

arangosh>db.complex.geo("work").near(0,170).limit(5).toArray();

SimpleQueries

679


Relatedtopics

OtherArangoDBgeographicfeaturesaredescribedin:

AQLGeofunctionsGeoindexes

SimpleQueries

680

Fulltextqueries

ItisrecommendedtouseAQLinstead,seeFulltextfunctions.

ArangoDBallowstorunqueriesontextcontainedindocumentattributes.Tousethis,afulltextindexmustbedefinedfortheattributeofthecollectionthatcontainsthetext.Creatingtheindexwillparsethetextinthespecifiedattributeforalldocumentsofthecollection.Onlydocumentswillbeindexedthatcontainatextualvalueintheindexedattribute.Forsuchdocuments,thetextvaluewillbeparsed,andtheindividualwordswillbeinsertedintothefulltextindex.

Whenafulltextindexexists,itcanbequeriedusingafulltextquery.

Fulltext

queriesthefulltextindexcollection.fulltext(attribute,query)

Thefulltextsimplequeryfunctionsperformsafulltextsearchonthespecifiedattributeandthespecifiedquery.

Detailsaboutthefulltextquerysyntaxcanbefoundbelow.

Note:thefulltextsimplequeryfunctionisdeprecatedasofArangoDB2.6.ThefunctionmayberemovedinfutureversionsofArangoDB.ThepreferredwayforexecutingfulltextqueriesistouseanAQLqueryusingtheFULLTEXTAQLfunctionasfollows:

FORdocINFULLTEXT(@@collection,@attributeName,@queryString,@limit)

RETURNdoc

Examples

arangosh>db.emails.ensureFulltextIndex("content");

arangosh>db.emails.save({content:

........>"HelloAlice,howareyoudoing?Regards,Bob"});

arangosh>db.emails.save({content:

........>"HelloCharlie,doAliceandBobknowaboutit?"});

arangosh>db.emails.save({content:"Ithinktheydon'tknow.Regards,Eve"});

arangosh>db.emails.fulltext("content","charlie,|eve").toArray();


FulltextSyntax:

Inthesimplestform,afulltextquerycontainsjustthesoughtword.Ifmultiplesearchwordsaregiveninaquery,theyshouldbeseparatedbycommas.AllsearchwordswillbecombinedwithalogicalANDbydefault,andonlysuchdocumentswillbereturnedthatcontainallsearchwords.Thisdefaultbehaviorcanbechangedbyprovidingtheextracontrolcharactersinthefulltextquery,whichare:

+:logicalAND(intersection)|:logicalOR(union)-:negation(exclusion)

Examples:

"banana":searchesfordocumentscontaining"banana""banana,apple":searchesfordocumentscontainingboth"banana"AND"apple""banana,|orange":searchesfordocumentscontainingeither"banana"OR"orange"ORboth"banana,-apple":searchesfordocumentsthatcontains"banana"butNOT"apple".

Logicaloperatorsareevaluatedfromlefttoright.

SimpleQueries

681

Eachsearchwordcanoptionallybeprefixedwithcomplete:orprefix:,withcomplete:beingthedefault.Thisallowssearchingforcompletewordsorforwordprefixes.Suffixsearchesoranyotherformsarepartial-wordmatchingarecurrentlynotsupported.

Examples:

"complete:banana":searchesfordocumentscontainingtheexactword"banana""prefix:head":searchesfordocumentswithwordsthatstartwithprefix"head""prefix:head,banana":searchesfordocumentscontainwordsstartingwithprefix"head"andthatalsocontaintheexactword"banana".

Completematchandprefixsearchoptionscanbecombinedwiththelogicaloperators.

Pleasenotethatonlywordswithaminimumlengthwillgetindexed.Thisminimumlengthcanbedefinedwhencreatingthefulltextindex.Forwordstokenization,thelibicutextboundaryanalysisisused,whichtakesintoaccountthedefaultasdefinedatserverstartup(--server.default-languagestartupoption).Generally,thewordboundaryanalysiswillfilteroutpunctuationbutwillnotdomuchmore.

Especiallynowordnormalization,stemming,orsimilarityanalysiswillbeperformedwhenindexingorsearching.Ifanyofthesefeaturesisrequired,itissuggestedthattheuserdoesthetextnormalizationontheclientside,andprovidesforeachdocumentanextraattributecontainingjustacomma-separatedlistofnormalizedwords.Thisattributecanthenbeindexedwithafulltextindex,andtheusercansendfulltextqueriesforthisindex,withthefulltextqueriesalsocontainingthestemmedornormalizedversionsofwordsasrequiredbytheuser.

SimpleQueries

682

ArangoDB'sActions

ItisrecommendedtouseFoxxinstead.

IntroductiontoUserActions

InsomewaysthecommunicationlayeroftheArangoDBserverbehaveslikeaWebserver.UnlikeaWebserver,itnormallyrespondstoHTTPrequestsbydeliveringJSONobjects.Remember,documentsinthedatabasearejustJSONobjects.So,mostofthetimetheHTTPresponsewillcontainaJSONdocumentfromthedatabaseasbody.YoucanextractthedocumentsstoredinthedatabaseusingHTTPGET.YoucanstoredocumentsusingHTTPPOST.

However,thereissomethingmore.Youcanwritesmallsnippets-socalledactions-toextendthedatabase.TheideaofactionsisthatsometimesitisbettertostorepartsofthebusinesslogicwithinArangoDB.

Thesimplestexampleistheageofaperson.Assumeyoustoreinformationaboutpeopleinyourdatabase.Itisananti-patterntostoretheage,becauseitchangeseverynowandthen.Therefore,younormallystorethebirthdayandlettheclientdecidewhattodowithit.However,ifyouhavemanydifferentclients,itmightbeeasiertoenrichthepersondocumentwiththeageusingactionsonceontheserverside.

Or,forinstance,ifyouwanttoapplysomestatisticstolargedata-setsandyoucannoteasilyexpressthisasquery.Youcandefineaactioninsteadoftransferringthewholedatatotheclientanddothecomputationontheclient.

Actionsarealsousefulifyouwanttorestrictandfilterdataaccordingtosomecomplexpermissionsystem.

TheArangoDBservercandeliverallkindsofinformation,JSONbeingonlyonepossibleformat.YoucanalsogenerateHTMLorimages.However,aWebserverisnormallybettersuitedforthetaskasitalsoimplementsvariouscachingstrategies,languageselection,compressionandsoon.Havingsaidthat,therearestillsituationswhereitmightbesuitabletousetheArangoDBtodeliverHTMLpages-staticordynamic.Asimpleexampleisthebuilt-inadministrationinterface.YoucanaccessitusinganymodernbrowserandthereisnoneedforaseparateApacheorIIS.

IngeneralyouwilluseFoxxtoeasilyextendthedatabasewithbusinesslogic.Foxxprovidesansimpletouseinterfacetoactions.

Thefollowingsectionswillexplainthelow-levelactionswithinArangoDBonwhichFoxxisbuiltandshowhowtodefinethem.TheexamplesstartwithdeliveringstaticHTMLpages-evenifthisisnottheprimaryuse-caseforactions.ThelatersectionswillthenshowyouhowtocodesomepiecesofyourbusinesslogicandreturnJSONobjects.

TheinterfaceislooselymodeledaftertheJavaScriptclassesforHTTPrequestandresponsesfoundinnode.jsandthemiddleware/routingaspectsofconnect.jsandexpress.js.

Notethatunlikenode.js,ArangoDBismulti-threadedandthereisnoeasywaytosharestatebetweenqueriesinsidetheJavaScriptengine.Ifsuchstateinformationisrequired,youneedtousethedatabaseitself.

Actions

683

AHelloWorldExample

TheclientAPIorbrowsersendsaHTTPrequesttotheArangoDBserverandtheserverreturnsaHTTPresponsetotheclient.AHTTPrequestconsistsofamethod,normallyGETorPOSTwhenusingabrowser,andarequestpathlike/hello/world.ForarealWebserverthereareazillionofotherthingtoconsider,wewillignorethisforthemoment.TheHTTPresponsecontainsacontenttype,describinghowtointerpretthereturneddata,andthedataitself.

Inthefollowingexample,wewanttodefineanactioninArangoDB,sothattheserverreturnstheHTMLdocument

<html>

<body>

HelloWorld

</body>

</html>

ifaskedGET/hello/world.

Theserverneedstoknowwhatfunctiontocallorwhatdocumenttodeliverifitreceivesarequest.Thisiscalledrouting.AlltheroutinginformationofArangoDBisstoredinacollection_routing.Eachentryinthiscollectionsdescribeshowtodealwithaparticularrequestpath.

Fortheaboveexample,addthefollowingdocumenttothe_routingcollection:

arangosh>db._routing.save({

........>url:{

........>match:"/hello/world"

........>},

........>content:{

........>contentType:"text/html",

........>body:"<html><body>HelloWorld</body></html>"

........>}

........>});

showexecutionresultsInordertoactivatethenewrouting,youmusteitherrestarttheserverorcalltheinternalreloadfunction.

arangosh>require("internal").reloadRouting()

Nowusethebrowserandaccesshttp://localhost:8529/hello/world

YoushouldseetheHelloWorldinourbrowser:

shell>curl--dump-http://localhost:8529/hello/world

HTTP/1.1200OK

content-type:text/html

x-content-type-options:nosniff

"HelloWorld"

MatchingaURLTherearealotofoptionsfortheurlattribute.Ifyoudefinedifferentroutingforthesamepath,thenthefollowingsimpleruleisappliedinordertodeterminewhichmatchwins:Iftherearetwomatches,thenthemorespecificwins.I.e,ifthereisawildcardmatchandanexactmatch,theexactmatchispreferred.Ifthereisashortandalongmatch,thelongermatchwins.

Actions

684

ExactMatch

Ifthedefinitionis

{

url:{

match:"/hello/world"

}

}

thenthematchmustbeexact.Onlytherequestfor/hello/worldwillmatch,everythingelse,e.g./hello/world/myor/hello/world2,willnotmatch.

Thefollowingdefinitionisashort-cutforanexactmatch.

{

url:"/hello/world"

}

Note:WhilethetwodefinitionswillresultinthesameURLmatching,thereisasubtledifferencebetweenthem:

Theformerdefinition(definingurlasanobjectwithamatchattribute)willresultintheURLbeingaccessibleviaallsupportedHTTPmethods(e.g.GET,POST,PUT,DELETE,...),whereasthelatterdefinition(providingastringurlattribute)willresultintheURLbeingaccessibleviaHTTPGETandHTTPHEADonly,withallotherHTTPmethodsbeingdisabled.CallingaURLwithanunsupportedordisabledHTTPmethodwillresultinanHTTP501(notimplemented)error.

PrefixMatch

Ifthedefinitionis

{

url:{

match:"/hello/world/*"

}

}

thenthematchcanbeaprefixmatch.Therequestsfor/hello/world,/hello/world/my,and/hello/world/how/are/youwillallmatch.However/hello/world2doesnotmatch.PrefixmatcheswithinaURLpart,i.e./hello/world*,arenotallowed.Thewildcardmustoccurattheend,i.e.

/hello/*/world

isalsodisallowed.

Ifyoudefinetworoutes

{url:{match:"/hello/world/*"}}

{url:{match:"/hello/world/emil"}}

thenthesecondroutewillbeusedfor/hello/world/emilbecauseitismorespecific.

ParameterizedMatch

Aparameterizedmatchissimilartoaprefixmatch,buttheparametersarealsoallowedinsidetheURLpath.

Ifthedefinitionis

{

url:{

match:"/hello/:name/world"

}

}

Actions

685

thentheURLmusthavethreeparts,thefirstpartbeinghelloandthethirdpartworld.Forexample,/hello/emil/worldwillmatch,while/hello/emil/meyer/worldwillnot.

ConstraintMatch

Aconstraintmatchissimilartoaparameterizedmatch,buttheparameterscancarryconstraints.

Ifthedefinitionis

{

url:{

match:"/hello/:name/world",

constraint:{

name:"/[a-z]+/"

}

}

}

thentheURLmusthavethreeparts,thefirstpartbeinghelloandthethirdpartworld.Thesecondpartmustbealllowercase.

ItispossibletousemorethenoneconstraintforthesameURLpart.

{

url:{

match:"/hello/:name|:id/world",

constraint:{

name:"/[a-z]+/",id:"/[0-9]+/"

}

}

}

OptionalMatch

Anoptionalmatchissimilartoaparameterizedmatch,butthelastparameterisoptional.

Ifthedefinitionis

{

url:{

match:"/hello/:name?",

constraint:{

name:"/[a-z]+/"

}

}

}

thentheURL/helloand/hello/emilwillmatch.

Ifthedefinitionsare

{url:{match:"/hello/world"}}

{url:{match:"/hello/:name",constraint:{name:"/[a-z]+/"}}}

{url:{match:"/hello/*"}}

thentheURL/hello/worldwillbematchedbythefirstroute,becauseitisthemostspecific.TheURL/hello/youwillbematchedbythesecondroute,becauseitismorespecificthantheprefixmatch.

MethodRestriction

YoucanrestrictthematchtospecificHTTPmethods.

Ifthedefinitionis

{

url:{

Actions

686

match:"/hello/world",

methods:["post","put"]

}

}

thenonlyHTTPPOSTandPUTrequestswillmatch.CallingwithadifferentHTTPmethodwillresultinanHTTP501error.

Pleasenotethatifurlisdefinedasasimplestring,thenonlytheHTTPmethodsGETandHEADwillbeallowed,anallothermethodswillbedisabled:

{

url:"/hello/world"

}

MoreonMatching

Rememberthatthemorespecificmatchwins.

Amatchwithoutparameterorwildcardismorespecificthanamatchwithparametersorwildcard.Amatchwithparameterismorespecificthanamatchwithawildcard.Ifthereismorethanoneparameter,specificityisappliedfromlefttoright.

Considerthefollowingdefinitions


........>url:{match:"/hello/world"},

........>content:{contentType:"text/plain",body:"MatchNo1"}});


........>url:{match:"/hello/:name",constraint:{name:"/[a-z]+/"}},



........>url:{match:"/:something/world"},



........>url:{match:"/hi/*"},



showexecutionresultsThen


HTTP/1.1200OK

content-type:text/plain


"MatchNo1"

shell>curl--dump-http://localhost:8529/hello/emil

HTTP/1.1200OK



"MatchNo2"

shell>curl--dump-http://localhost:8529/your/world

HTTP/1.1200OK

Actions

687



"MatchNo3"

shell>curl--dump-http://localhost:8529/hi/you

HTTP/1.1200OK



"MatchNo4"

Youcanwritethefollowingdocumentintothe_routingcollectiontotesttheaboveexamples.

{

routes:[

{url:{match:"/hello/world"},content:"route1"},

{url:{match:"/hello/:name|:id",constraint:{name:"/[a-z]+/",id:"/[0-9]+/"}},content:"route2"},

{url:{match:"/:something/world"},content:"route3"},

{url:{match:"/hello/*"},content:"route4"},

]

}

Actions

688

AHelloWorldExampleforJSON

Ifyouchangetheexampleslightly,thenaJSONobjectwillbedelivered.


........>url:"/hello/json",

........>content:{

........>contentType:"application/json",

........>body:'{"hello":"world"}'

........>}

........>});


showexecutionresultsAgaincheckwithyourbrowserorcURLhttp://localhost:8529/hello/json

Dependingonyourbrowserandinstalledadd-onsyouwilleitherseetheJSONobjectoradownloaddialog.IfyourbrowserwantstoopenanexternalapplicationtodisplaytheJSONobject,youcanchangethecontentTypeto"text/plain" fortheexample.Thismakesiteasiertochecktheexampleusingabrowser.Orusecurltoaccesstheserver.

shell>curl--dump-http://localhost:8529/hello/json

HTTP/1.1200OK

content-type:application/json;charset=utf-8


{

"hello":"world"

}

DeliveringContent

Therearealotofdifferentwaysonhowtodelivercontent.Wehavealreadyseenthesimplestone,wherestaticcontentisdelivered.Thefun,however,startswhendeliveringdynamiccontent.

StaticContent

Youcanspecifyabodyandacontent-type.


........>url:"/hello/contentType",

........>content:{

........>contentType:"text/html",

........>body:"<html><body>HelloWorld</body></html>"

........>}

........>});



shell>curl--dump-http://localhost:8529/hello/contentType

Actions

689

http://localhost:8529/hello/json

HTTP/1.1200OK



"HelloWorld"

Ifthecontenttypeistext/plainthenyoucanusetheshort-cut

{

content:"HelloWorld"

}

ASimpleAction

Thesimplestdynamicactionis:

{

action:{

do:"@arangodb/actions/echoRequest"

}

}

Itisnotadvisabletostorefunctionsdirectlyintheroutingtable.Itisbettertocallfunctionsdefinedinmodules.IntheaboveexamplethefunctioncanbeaccessedfromJavaScriptas:

require("@arangodb/actions").echoRequest

ThefunctionechoRequestispre-defined.Ittakestherequestobjectsandechositintheresponse.

Thesignatureofsuchafunctionmustbe

function(req,res,options,next)

Examples


........>url:"/hello/echo",

........>action:{

........>do:"@arangodb/actions/echoRequest"

........>}

........>});

showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo

Youshouldseesomethinglike

arangosh>arango.GET("/hello/echo")


Therequestmightcontainpath,prefix,suffix,andurlParametersattributes.pathisthecompletepathassuppliedbytheuserandalwaysavailable.Ifaprefixwasmatched,thenthisprefixisstoredintheattributeprefixandtheremainingURLpartsarestoredasanarrayinsuffix.Ifoneormoreparameterswerematched,thentheparametervaluesarestoredinurlParameters.

Actions

690

Forexample,iftheurldescriptionis

{

url:{

match:"/hello/:name/:action"

}

}

andyourequestthepath/hello/emil/jump,thentherequestobjectwillcontainthefollowingattribute

urlParameters:{

name:"emil",

action:"jump"

}

ActionController

Asanalternativetothesimpleaction,youcanusecontrollers.Acontrollerisamodule,definesthefunctionget,put,post,delete,head,patch.Ifarequestofthecorrespondingtypeismatched,thefunctionwillbecalled.

Examples



........>action:{

........>controller:"@arangodb/actions/echoController"

........>}

........>});

showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo:



PrefixActionController

Thecontrollerisselectedwhenthedefinitionisread.Thereisamoreflexible,butslowerandmaybeinsecurevariant,theprefixcontroller.

Assumethattheurlisaprefixmatch

{

url:{

match:/hello/*"

}

}

Youcanuse

{

action:{

prefixController:"@arangodb/actions"

}

}

todefineaprefixcontroller.IftheURL/hello/echoControllerisgiven,thenthemodule@arangodb/actions/echoControllerisused.

Ifyouuseaprefixcontroller,youshouldmakecertainthatnounwantedactionsareavailableundertheprefix.

Actions

691

Thedefinition

{

action:"@arangodb/actions"

}

isashort-cutforaprefixcontrollerdefinition.

FunctionAction

Youcanalsostoreafunctiondirectlyintheroutingtable.

Examples



........>action:{

........>callback:"function(req,res){res.statusCode=200;res.body='Hello'}"

........>}

........>});


arangosh>arango.GET("hello/echo")

arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/hello/echo'REMOVEroute

in_routing")

[objectArangoQueryCursor,count:0,cached:false,hasMore:false]


RequestsandResponses

Thecontrollermustdefinehandlerfunctionswhichtakearequestobjectandfilltheresponseobject.

AverysimpleexampleisthefunctionechoRequestdefinedinthemodule@arangodb/actions.

function(req,res,options,next){

varresult;

result={request:req,options:options};

res.responseCode=exports.HTTP_OK;

res.contentType="application/json";

res.body=JSON.stringify(result);

}

Installitvia:


........>url:"/echo",

........>action:{


........>}

........>})

showexecutionresultsReloadtheroutingandcheckhttp://127.0.0.1:8529/hello/echo

Youshouldseesomethinglike

Actions

692


arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/hello/echo'REMOVEroute

in_routing")


showexecutionresultsYoumayalsopassoptionstothecalledfunction:


........>url:"/echo",

........>action:{

........>do:"@arangodb/actions/echoRequest",

........>options:{

........>"Hello":"World"

........>}

........>}

........>});

showexecutionresultsYounowseetheoptionsintheresult:

arangosh>arango.GET("/echo")

arangosh>db._query("FORrouteIN_routingFILTERroute.url=='/echo'REMOVEroutein

_routing")



Actions

693

ModifyingRequestandResponse

Aswe'veseeninthepreviousexamples,actionsgetcalledwiththerequestandresponseobjects(namedreqandresintheexamples)passedasparameterstotheirhandlerfunctions.

ThereqobjectcontainstheincomingHTTPrequest,whichmightormightnothavebeenmodifiedbyapreviousaction(ifactionswerechained).

Ahandlercanmodifytherequestobjectinplaceifdesired.Thismightbeusefulwhenwritingmiddleware(seebelow)thatisusedtointerceptincomingrequests,modifythemandpassthemtotheactualhandlers.

Whilemodifyingtherequestobjectmightnotbethatrelevantfornon-middlewareactions,modifyingtheresponseobjectdefinitelyis.Modifyingtheresponseobjectisanaction'sonlywaytoreturndatatothecalleroftheaction.

We'vealreadyseenhowtosettheHTTPstatuscode,thecontenttype,andtheresultbody.Theresobjecthasthefollowingpropertiesforthese:

contentType:MIMEtypeofthebodyasdefinedintheHTTPstandard(e.g.text/html,text/plain,application/json,...)responsecode:theHTTPstatuscodeoftheresponseasdefinedintheHTTPstandard.Commonvaluesforactionsthatsucceedare200or201.PleaserefertotheHTTPstandardformoreinformation.body:theactualresponsedata

Tosetormodifyarbitraryheadersoftheresponseobject,theheaderspropertycanbeused.Forexample,toaddauser-definedheadertotheresponse,thefollowingcodewilldo:

res.headers=res.headers||{};//headersmightormightnotbepresent

res.headers['X-Test']='someValue';//setheaderX-Testto"someValue"

ThiswillsettheadditionalHTTPheaderX-TesttovaluesomeValue.Otherheaderscanbesetaswell.NotethatArangoDBmightchangethecaseoftheheadernamestolowercasewhenassemblingtheoverallresponsethatissenttothecaller.

ItisnotnecessarytoexplicitlysetaContent-LengthheaderfortheresponseasArangoDBwillcalculatethecontentlengthautomaticallyandaddthisheaderitself.ArangoDBmightalsoaddaConnectionheaderitselftohandleHTTPkeep-alive.

ArangoDBalsosupportsautomatictransformationofthebodydatatoanotherformat.Currently,theonlysupportedtransformationsarebase64-encodingandbase64-decoding.Usingthetransformations,anactioncancreateabase64encodedbodyandstillletArangoDBsendthenon-encodedversion,forexample:

res.body='VGhpcyBpcyBhIHRlc3Q=';

res.transformations=res.transformations||[];//initialize

res.transformations.push('base64decode');//willbase64decodetheresponsebody

WhenArangoDBprocessestheresponse,itwillbase64-decodewhat'sinres.bodyandsettheHTTPheaderContent-Encoding:binary.Theoppositecanbeachievedwiththebase64encodetransformation:ArangoDBwillthenautomaticallybase64-encodethebodyandsetaContent-Encoding:base64HTTPheader.

WritingdynamicactionhandlersTowriteyourowndynamicactionhandlers,youmustputthemintomodules.

Modulesareameansoforganizingactionhandlersandmakingthemloadableunderspecificnames.

Tostart,we'lldefineasimpleactionhandlerinamodule/ownTest:

arangosh>db._modules.save({

........>path:"/db:/ownTest",

........>content:

........>"exports.do=function(req,res,options,next){"+

........>"res.body='test';"+

Actions

694

........>"res.responseCode=200;"+

........>"res.contentType='text/plain';"+

........>"};"

........>});

showexecutionresultsThisdoesnothingbutregisteradoactionhandlerinamodule/ownTest.Theactionhandlerisnotyetcallable,butmustbemappedtoaroutefirst.Tomaptheactiontotheroute/ourtest,executethefollowingcommand:


........>url:"/ourtest",

........>action:{

........>controller:"db://ownTest"

........>}

........>});


showexecutionresultsNowusethebrowserorcURLandaccesshttp://localhost:8529/ourtest:

shell>curl--dump-http://localhost:8529/ourtest

HTTP/1.1200OK



"test"

Youwillseethatthemodule'sdofunctionhasbeenexecuted.

AWordaboutCachingSometimesitmightseemthatyourchangedonottakeeffect.Inthiscasetheculpritcouldbetheroutingcaches:

Theroutingcachestorestheroutinginformationcomputedfromthe_routingcollection.Wheneveryouchangethiscollectionmanually,youneedtocall


inordertorebuildthecache.

AdvancedUsagesFordetailedinformationseethereferencemanual.

Redirects

Usethefollowingforapermanentredirect:


........>url:"/redirectMe",

........>action:{

........>do:"@arangodb/actions/redirectRequest",

........>options:{

Actions

695

http://localhost:8529/ourtest

........>permanently:true,

........>destination:"/somewhere.else/"

........>}

........>}

........>});



shell>curl--dump-http://localhost:8529/redirectMe

HTTP/1.1301MovedPermanently



location:/somewhere.else/

"<html><head><title>Moved</title></head><body><h1>Moved</h1><p>Thispagehasmovedto<a

href=\"/somewhere.else/\">/somewhere.else/</a>.</p></body></html>"

RoutingBundles

Insteadofaddingallroutesforpackageseparately,youcanspecifyabundle:


........>routes:[

........>{

........>url:"/url1",

........>content:"route1"

........>},

........>{

........>url:"/url2",


........>},

........>{

........>url:"/url3",


........>}

........>]

........>});



shell>curl--dump-http://localhost:8529/url2

HTTP/1.1200OK



"route2"

shell>curl--dump-http://localhost:8529/url3

HTTP/1.1200OK



Actions

696

"route3"

Theadvantageis,thatyoucanputallyourroutesintoonedocumentanduseacommonprefix.


........>urlPrefix:"/test",

........>routes:[

........>{

........>url:"/url1",


........>},

........>{

........>url:"/url2",


........>},

........>{

........>url:"/url3",


........>}

........>]

........>});


showexecutionresultswilldefinetheURL/test/url1,/test/url2,and/test/url3:

shell>curl--dump-http://localhost:8529/test/url1

HTTP/1.1200OK



"route1"


HTTP/1.1200OK



"route2"


HTTP/1.1200OK



"route3"

WritingMiddleware

Assume,youwanttologeveryrequestinyournamespacetotheconsole.(ifArangoDBisrunningasadaemon,thiswillendupinthelogfile).InthiscaseyoucaneasilydefineanactionfortheURL/subdirectory.Thisactionsimplylogstherequests,callsthenextinline,andlogstheresponse:

arangosh>db._modules.save({

Actions

697

........>path:"/db:/OwnMiddlewareTest",

........>content:

........>"exports.logRequest=function(req,res,options,next){"+

........>"console=require('console');"+

........>"console.log('receivedrequest:%s',JSON.stringify(req));"+

........>"next();"+

........>"console.log('producedresponse:%s',JSON.stringify(res));"+

........>"};"

........>});

showexecutionresultsThisfunctionwillnowbeavailableasdb://OwnMiddlewareTest/logRequest.YouneedtotellArangoDBthatitisshoulduseaprefixmatchandthattheshortestmatchshouldwininthiscase:


........>middleware:[

........>{

........>url:{

........>match:"/subdirectory/*"

........>},

........>action:{

........>do:"db://OwnMiddlewareTest/logRequest"

........>}

........>}

........>]

........>});

showexecutionresultsWhenyoucallnext()inthataction,thenextspecificroutingwillbeusedfortheoriginalURL.EvenifyoumodifytheURLintherequestobjectreq,thiswillnotcausethenext()tojumptotheroutingdefinedforthisnextURL.IfproceedsoccurringtheoriginURL.However,ifyouusenext(true),theroutingwillstopandrequesthandlingisstartedwiththenewURL.Youmustensurethatnext(true)isnevercalledwithoutmodifyingtheURLintherequestobjectreq.Otherwiseanendlessloopwilloccur.

Nowweaddsomeothersimpleroutingstotestallthis:


........>url:"/subdirectory/ourtest/1",

........>action:{


........>}

........>});



........>action:{


........>}

........>});



........>action:{


........>}

........>});



Actions

698

Thenwesendsomecurlrequeststothesesampleroutes:

shell>curl--dump-http://localhost:8529/subdirectory/ourtest/1

HTTP/1.1200OK



showresponsebodyandtheconsole(and/orthelogfile)willshowrequestsandreplies.Notethatloggingdoesn'twarrantthesequenceinwhichtheselineswillappear.

ApplicationDeploymentUsingsingleroutesorbundlescanbebecomeabitmessyinlargeapplications.Kaerushaswrittenadeploymenttoolinnode.js.

NotethatthereisalsoFoxxforbuildingapplicationswithArangoDB.

CommonPitfallswhenusingActions

Caching

IfyoumadeanychangestotheroutingbutthechangesdoesnothaveanyeffectwhencallingthemodifiedactionsURL,youmighthavebeenhitbysomecachingissues.

Afteranymodificationtotheroutingoractions,itisthusrecommendedtomakethechanges"live"bycallingthefollowingfunctionsfromwithinarangosh:

Youmightalsobeaffectedbyclient-sidecaching.BrowserstendtocachecontentandalsoredirectionURLs.Youmightneedtoclearordisablethebrowsercacheinsomecasestoseeyourchangesineffect.

Datatypes

Whenprocessingtherequestdatainanaction,pleasebeawarethatthedatatypeofallqueryparametersisstring.ThisisbecausethewholeURLisastringandwhentheindividualpartsareextracted,theywillalsobestrings.

Forexample,whencallingtheURLhttp://localhost:8529/hello/world?value=5

theparametervaluewillhaveavalueof(string)5,not(number)5.ThismightbetroublesomeifyouuseJavaScript's===operatorwhencheckingrequestparametervalues.

ThesameproblemoccurswithincomingHTTPheaders.WhensendingthefollowingheaderfromaclienttoArangoDB

X-My-Value:5

thentheheaderX-My-Valuewillhaveavalueof(string)5andnot(number)5.

404NotFound

IfyoudefinedaURLintheroutingandtheURLisaccessiblefineviaHTTPGETbutreturnsanHTTP501(notimplemented)forotherHTTPmethodssuchasPOST,PUTorDELETE,thenyoumighthavebeenhitbysomedefaults.

Bydefault,URLsdefinedlikethis(simplestringurlattribute)areaccessibleviaHTTPGETandHEADonly.TomakesuchURLsaccessibleviaotherHTTPmethods,extendtheURLdefinitionwiththemethodsattribute.

Forexample,thisdefinitiononlyallowsaccessviaGETandHEAD:

Actions

699

https://github.com/kaerus/arangodep

{

url:"/hello/world"

}

whereasthisdefinitionallowsHTTPGET,POST,andPUT:


........>url:{

........>match:"/hello/world",

........>methods:["get","post","put"]

........>},

........>action:{


........>}

........>});




HTTP/1.1200OK



showresponsebodyTheformerdefinition(definingurlasanobjectwithamatchattribute)willresultintheURLbeingaccessibleviaallsupportedHTTPmethods(e.g.GET,POST,PUT,DELETE,...),whereasthelatterdefinition(providingastringurlattribute)willresultintheURLbeingaccessibleviaHTTPGETandHTTPHEADonly,withallotherHTTPmethodsbeingdisabled.CallingaURLwithanunsupportedordisabledHTTPmethodwillresultinanHTTP404error.

Actions

700


Generalerrors

0-ERROR_NO_ERRORNoerrorhasoccurred.

1-ERROR_FAILEDWillberaisedwhenageneralerroroccurred.

2-ERROR_SYS_ERRORWillberaisedwhenoperatingsystemerroroccurred.

3-ERROR_OUT_OF_MEMORYWillberaisedwhenthereisamemoryshortage.

4-ERROR_INTERNALWillberaisedwhenaninternalerroroccurred.

5-ERROR_ILLEGAL_NUMBERWillberaisedwhenanillegalrepresentationofanumberwasgiven.

6-ERROR_NUMERIC_OVERFLOWWillberaisedwhenanumericoverflowoccurred.

7-ERROR_ILLEGAL_OPTIONWillberaisedwhenanunknownoptionwassuppliedbytheuser.

8-ERROR_DEAD_PIDWillberaisedwhenaPIDwithoutalivingprocesswasfound.

9-ERROR_NOT_IMPLEMENTEDWillberaisedwhenhittinganunimplementedfeature.

10-ERROR_BAD_PARAMETERWillberaisedwhentheparameterdoesnotfulfilltherequirements.

11-ERROR_FORBIDDENWillberaisedwhenyouaremissingpermissionfortheoperation.

12-ERROR_OUT_OF_MEMORY_MMAPWillberaisedwhenthereisamemoryshortage.

13-ERROR_CORRUPTED_CSVWillberaisedwhenencounteringacorruptcsvline.

14-ERROR_FILE_NOT_FOUNDWillberaisedwhenafileisnotfound.

15-ERROR_CANNOT_WRITE_FILEWillberaisedwhenafilecannotbewritten.

16-ERROR_CANNOT_OVERWRITE_FILEWillberaisedwhenanattemptismadetooverwriteanexistingfile.

17-ERROR_TYPE_ERRORWillberaisedwhenatypeerrorisunencountered.

18-ERROR_LOCK_TIMEOUTWillberaisedwhenthere'satimeoutwaitingforalock.

19-ERROR_CANNOT_CREATE_DIRECTORYWillberaisedwhenanattempttocreateadirectoryfails.


701

20-ERROR_CANNOT_CREATE_TEMP_FILEWillberaisedwhenanattempttocreateatemporaryfilefails.

21-ERROR_REQUEST_CANCELEDWillberaisedwhenarequestiscanceledbytheuser.

22-ERROR_DEBUGWillberaisedintentionallyduringdebugging.

25-ERROR_IP_ADDRESS_INVALIDWillberaisedwhenthestructureofanIPaddressisinvalid.

27-ERROR_FILE_EXISTSWillberaisedwhenafilealreadyexists.

28-ERROR_LOCKEDWillberaisedwhenaresourceoranoperationislocked.

29-ERROR_DEADLOCKWillberaisedwhenadeadlockisdetectedwhenaccessingcollections.

30-ERROR_SHUTTING_DOWNWillberaisedwhenacallcannotsucceedbecauseaservershutdownisalreadyinprogress.

31-ERROR_ONLY_ENTERPRISEWillberaisedwhenanenterprise-featureisrequestedfromthecommunityedition.

32-ERROR_RESOURCE_LIMITWillberaisedwhentheresourcesusedbyanoperationexceedtheconfiguredmaximumvalue.

HTTPerrorstatuscodes400-ERROR_HTTP_BAD_PARAMETERWillberaisedwhentheHTTPrequestdoesnotfulfilltherequirements.

401-ERROR_HTTP_UNAUTHORIZEDWillberaisedwhenauthorizationisrequiredbuttheuserisnotauthorized.

403-ERROR_HTTP_FORBIDDENWillberaisedwhentheoperationisforbidden.

404-ERROR_HTTP_NOT_FOUNDWillberaisedwhenanURIisunknown.

405-ERROR_HTTP_METHOD_NOT_ALLOWEDWillberaisedwhenanunsupportedHTTPmethodisusedforanoperation.

406-ERROR_HTTP_NOT_ACCEPTABLEWillberaisedwhenanunsupportedHTTPcontenttypeisusedforanoperation,orifarequestisnotacceptableforaleaderorfollower.

412-ERROR_HTTP_PRECONDITION_FAILEDWillberaisedwhenapreconditionforanHTTPrequestisnotmet.

500-ERROR_HTTP_SERVER_ERRORWillberaisedwhenaninternalserverisencountered.

503-ERROR_HTTP_SERVICE_UNAVAILABLEWillberaisedwhenaserviceistemporarilyunavailable.

504-ERROR_HTTP_GATEWAY_TIMEOUTWillberaisedwhenaservicecontactedbyArangoDBdoesnotrespondinatimelymanner.


702

HTTPprocessingerrors

600-ERROR_HTTP_CORRUPTED_JSONWillberaisedwhenastringrepresentationofaJSONobjectiscorrupt.

601-ERROR_HTTP_SUPERFLUOUS_SUFFICESWillberaisedwhentheURLcontainssuperfluoussuffices.

InternalArangoDBstorageerrorsForerrorsthatoccurbecauseofaprogrammingerror.

1000-ERROR_ARANGO_ILLEGAL_STATEInternalerrorthatwillberaisedwhenthedatafileisnotintherequiredstate.

1002-ERROR_ARANGO_DATAFILE_SEALEDInternalerrorthatwillberaisedwhentryingtowritetoadatafile.

1004-ERROR_ARANGO_READ_ONLYInternalerrorthatwillberaisedwhentryingtowritetoaread-onlydatafileorcollection.

1005-ERROR_ARANGO_DUPLICATE_IDENTIFIERInternalerrorthatwillberaisedwhenaidentifierduplicateisdetected.

1006-ERROR_ARANGO_DATAFILE_UNREADABLEInternalerrorthatwillberaisedwhenadatafileisunreadable.

1007-ERROR_ARANGO_DATAFILE_EMPTYInternalerrorthatwillberaisedwhenadatafileisempty.

1008-ERROR_ARANGO_RECOVERYWillberaisedwhenanerroroccurredduringWALlogfilerecovery.

1009-ERROR_ARANGO_DATAFILE_STATISTICS_NOT_FOUNDWillberaisedwhenarequireddatafilestatisticsobjectwasnotfound.

ExternalArangoDBstorageerrors

Forerrorsthatoccurbecauseofanoutsideevent.

1100-ERROR_ARANGO_CORRUPTED_DATAFILEWillberaisedwhenacorruptionisdetectedinadatafile.

1101-ERROR_ARANGO_ILLEGAL_PARAMETER_FILEWillberaisedifaparameterfileiscorruptedorcannotberead.

1102-ERROR_ARANGO_CORRUPTED_COLLECTIONWillberaisedwhenacollectioncontainsoneormorecorrupteddatafiles.

1103-ERROR_ARANGO_MMAP_FAILEDWillberaisedwhenthesystemcallmmapfailed.

1104-ERROR_ARANGO_FILESYSTEM_FULLWillberaisedwhenthefilesystemisfull.

1105-ERROR_ARANGO_NO_JOURNALWillberaisedwhenajournalcannotbecreated.

1106-ERROR_ARANGO_DATAFILE_ALREADY_EXISTSWillberaisedwhenthedatafilecannotbecreatedorrenamedbecauseafileofthesamenamealreadyexists.

1107-ERROR_ARANGO_DATADIR_LOCKEDWillberaisedwhenthedatabasedirectoryislockedbyadifferentprocess.


703

1108-ERROR_ARANGO_COLLECTION_DIRECTORY_ALREADY_EXISTSWillberaisedwhenthecollectioncannotbecreatedbecauseadirectoryofthesamenamealreadyexists.

1109-ERROR_ARANGO_MSYNC_FAILEDWillberaisedwhenthesystemcallmsyncfailed.

1110-ERROR_ARANGO_DATADIR_UNLOCKABLEWillberaisedwhentheservercannotlockthedatabasedirectoryonstartup.

1111-ERROR_ARANGO_SYNC_TIMEOUTWillberaisedwhentheserverwaitedtoolongforadatafiletobesyncedtodisk.

GeneralArangoDBstorageerrors

Forerrorsthatoccurwhenfulfillingauserrequest.

1200-ERROR_ARANGO_CONFLICTWillberaisedwhenupdatingordeletingadocumentandaconflicthasbeendetected.

1201-ERROR_ARANGO_DATADIR_INVALIDWillberaisedwhenanon-existingdatabasedirectorywasspecifiedwhenstartingthedatabase.

1202-ERROR_ARANGO_DOCUMENT_NOT_FOUNDWillberaisedwhenadocumentwithagivenidentifierorhandleisunknown.

1203-ERROR_ARANGO_COLLECTION_NOT_FOUNDWillberaisedwhenacollectionwiththegivenidentifierornameisunknown.

1204-ERROR_ARANGO_COLLECTION_PARAMETER_MISSINGWillberaisedwhenthecollectionparameterismissing.

1205-ERROR_ARANGO_DOCUMENT_HANDLE_BADWillberaisedwhenadocumenthandleiscorrupt.

1206-ERROR_ARANGO_MAXIMAL_SIZE_TOO_SMALLWillberaisedwhenthemaximalsizeofthejournalistoosmall.

1207-ERROR_ARANGO_DUPLICATE_NAMEWillberaisedwhenanameduplicateisdetected.

1208-ERROR_ARANGO_ILLEGAL_NAMEWillberaisedwhenanillegalnameisdetected.

1209-ERROR_ARANGO_NO_INDEXWillberaisedwhennosuitableindexforthequeryisknown.

1210-ERROR_ARANGO_UNIQUE_CONSTRAINT_VIOLATEDWillberaisedwhenthereisauniqueconstraintviolation.

1211-ERROR_ARANGO_VIEW_NOT_FOUNDWillberaisedwhenaviewwiththegivenidentifierornameisunknown.

1212-ERROR_ARANGO_INDEX_NOT_FOUNDWillberaisedwhenanindexwithagivenidentifierisunknown.

1213-ERROR_ARANGO_CROSS_COLLECTION_REQUESTWillberaisedwhenacross-collectionisrequested.

1214-ERROR_ARANGO_INDEX_HANDLE_BADWillberaisedwhenaindexhandleiscorrupt.

1216-ERROR_ARANGO_DOCUMENT_TOO_LARGEWillberaisedwhenthedocumentcannotfitintoanydatafilebecauseofitistoolarge.


704

1217-ERROR_ARANGO_COLLECTION_NOT_UNLOADEDWillberaisedwhenacollectionshouldbeunloaded,buthasadifferentstatus.

1218-ERROR_ARANGO_COLLECTION_TYPE_INVALIDWillberaisedwhenaninvalidcollectiontypeisusedinarequest.

1219-ERROR_ARANGO_VALIDATION_FAILEDWillberaisedwhenthevalidationofanattributeofastructurefailed.

1220-ERROR_ARANGO_ATTRIBUTE_PARSER_FAILEDWillberaisedwhenparsinganattributenamedefinitionfailed.

1221-ERROR_ARANGO_DOCUMENT_KEY_BADWillberaisedwhenadocumentkeyiscorrupt.

1222-ERROR_ARANGO_DOCUMENT_KEY_UNEXPECTEDWillberaisedwhenauser-defineddocumentkeyissuppliedforcollectionswithautokeygeneration.

1224-ERROR_ARANGO_DATADIR_NOT_WRITABLEWillberaisedwhentheserver'sdatabasedirectoryisnotwritableforthecurrentuser.

1225-ERROR_ARANGO_OUT_OF_KEYSWillberaisedwhenakeygeneratorrunsoutofkeys.

1226-ERROR_ARANGO_DOCUMENT_KEY_MISSINGWillberaisedwhenadocumentkeyismissing.

1227-ERROR_ARANGO_DOCUMENT_TYPE_INVALIDWillberaisedwhenthereisanattempttocreateadocumentwithaninvalidtype.

1228-ERROR_ARANGO_DATABASE_NOT_FOUNDWillberaisedwhenanon-existingdatabaseisaccessed.

1229-ERROR_ARANGO_DATABASE_NAME_INVALIDWillberaisedwhenaninvaliddatabasenameisused.

1230-ERROR_ARANGO_USE_SYSTEM_DATABASEWillberaisedwhenanoperationisrequestedinadatabaseotherthanthesystemdatabase.

1231-ERROR_ARANGO_ENDPOINT_NOT_FOUNDWillberaisedwhenthereisanattempttodeleteanon-existingendpoint.

1232-ERROR_ARANGO_INVALID_KEY_GENERATORWillberaisedwhenaninvalidkeygeneratordescriptionisused.

1233-ERROR_ARANGO_INVALID_EDGE_ATTRIBUTEwillberaisedwhenthe_fromor_tovaluesofanedgeareundefinedorcontainaninvalidvalue.

1234-ERROR_ARANGO_INDEX_DOCUMENT_ATTRIBUTE_MISSINGWillberaisedwhenanattempttoinsertadocumentintoanindexiscausedbyinthedocumentnothavingoneormoreattributeswhichtheindexisbuilton.

1235-ERROR_ARANGO_INDEX_CREATION_FAILEDWillberaisedwhenanattempttocreateanindexhasfailed.

1236-ERROR_ARANGO_WRITE_THROTTLE_TIMEOUTWillberaisedwhentheserveriswrite-throttledandawriteoperationhaswaitedtoolongfortheservertoprocessqueuedoperations.

1237-ERROR_ARANGO_COLLECTION_TYPE_MISMATCHWillberaisedwhenacollectionhasadifferenttypefromwhathasbeenexpected.

1238-ERROR_ARANGO_COLLECTION_NOT_LOADEDWillberaisedwhenacollectionisaccessedthatisnotyetloaded.


705

1239-ERROR_ARANGO_DOCUMENT_REV_BADWillberaisedwhenadocumentrevisioniscorruptorismissingwhereneeded.

CheckedArangoDBstorageerrors

Forerrorsthatoccurbutareanticipated.

1300-ERROR_ARANGO_DATAFILE_FULLWillberaisedwhenthedatafilereachesitslimit.

1301-ERROR_ARANGO_EMPTY_DATADIRWillberaisedwhenencounteringanemptyserverdatabasedirectory.

1302-ERROR_ARANGO_TRY_AGAINWillberaisedwhenanoperationshouldberetried.

1303-ERROR_ARANGO_BUSYWillberaisedwhenstorageengineisbusy.

1304-ERROR_ARANGO_MERGE_IN_PROGRESSWillberaisedwhenstorageenginehasadatafilemergeinprogressandcannotcompletetheoperation.

1305-ERROR_ARANGO_IO_ERRORWillberaisedwhenstorageengineencountersanI/Oerror.

ArangoDBreplicationerrors

1400-ERROR_REPLICATION_NO_RESPONSEWillberaisedwhenthereplicationapplierdoesnotreceiveanyoranincompleteresponsefromthemaster.

1401-ERROR_REPLICATION_INVALID_RESPONSEWillberaisedwhenthereplicationapplierreceivesaninvalidresponsefromthemaster.

1402-ERROR_REPLICATION_MASTER_ERRORWillberaisedwhenthereplicationapplierreceivesaservererrorfromthemaster.

1403-ERROR_REPLICATION_MASTER_INCOMPATIBLEWillberaisedwhenthereplicationapplierconnectstoamasterthathasanincompatibleversion.

1404-ERROR_REPLICATION_MASTER_CHANGEWillberaisedwhenthereplicationapplierconnectstoadifferentmasterthanbefore.

1405-ERROR_REPLICATION_LOOPWillberaisedwhenthereplicationapplierisaskedtoconnecttoitselfforreplication.

1406-ERROR_REPLICATION_UNEXPECTED_MARKERWillberaisedwhenanunexpectedmarkerisfoundinthereplicationlogstream.

1407-ERROR_REPLICATION_INVALID_APPLIER_STATEWillberaisedwhenaninvalidreplicationapplierstatefileisfound.

1408-ERROR_REPLICATION_UNEXPECTED_TRANSACTIONWillberaisedwhenanunexpectedtransactionidisfound.

1410-ERROR_REPLICATION_INVALID_APPLIER_CONFIGURATIONWillberaisedwhentheconfigurationforthereplicationapplierisinvalid.

1411-ERROR_REPLICATION_RUNNINGWillberaisedwhenthereisanattempttoperformanoperationwhilethereplicationapplierisrunning.

1412-ERROR_REPLICATION_APPLIER_STOPPEDSpecialerrorcodeusedtoindicatethereplicationapplierwasstoppedbyauser.


706

1413-ERROR_REPLICATION_NO_START_TICKWillberaisedwhenthereplicationapplierisstartedwithoutaknownstarttickvalue.

1414-ERROR_REPLICATION_START_TICK_NOT_PRESENTWillberaisedwhenthereplicationapplierfetchesdatausingastarttick,butthatstarttickisnotpresentontheloggerserveranymore.

ERROR_REPLICATION_WRONG_CHECKSUM_FORMAT,1415,"thechecksumformatiswrong","Willberaisedwhentheformatofthechecksumiswrong")

1416-ERROR_REPLICATION_WRONG_CHECKSUMWillberaisedwhenanewbornfollowersubmitsawrongchecksum

1417-ERROR_REPLICATION_SHARD_NONEMPTYWillberaisedwhenashardisnotemptyandthefollowertriesashortcut

ArangoDBclustererrors1450-ERROR_CLUSTER_NO_AGENCYWillberaisedwhennoneoftheagencyserverscanbeconnectedto.

1451-ERROR_CLUSTER_NO_COORDINATOR_HEADERWillberaisedwhenaDBserverinaclusterreceivesaHTTPrequestwithoutacoordinatorheader.

1452-ERROR_CLUSTER_COULD_NOT_LOCK_PLANWillberaisedwhenacoordinatorinaclustercannotlockthePlanhierarchyintheagency.

1453-ERROR_CLUSTER_COLLECTION_ID_EXISTSWillberaisedwhenacoordinatorinaclustertriestocreateacollectionandthecollectionIDalreadyexists.

1454-ERROR_CLUSTER_COULD_NOT_CREATE_COLLECTION_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewcollectioninthePlanhierarchyintheagency.

1455-ERROR_CLUSTER_COULD_NOT_READ_CURRENT_VERSIONWillberaisedwhenacoordinatorinaclustercannotreadtheVersionentryintheCurrenthierarchyintheagency.

1456-ERROR_CLUSTER_COULD_NOT_CREATE_COLLECTIONWillberaisedwhenacoordinatorinaclusternoticesthatsomeDBServersreportproblemswhencreatingshardsforanewcollection.

1457-ERROR_CLUSTER_TIMEOUTWillberaisedwhenacoordinatorinaclusterrunsintoatimeoutforsomeclusterwideoperation.

1458-ERROR_CLUSTER_COULD_NOT_REMOVE_COLLECTION_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanentryforacollectioninthePlanhierarchyintheagency.

1459-ERROR_CLUSTER_COULD_NOT_REMOVE_COLLECTION_IN_CURRENTWillberaisedwhenacoordinatorinaclustercannotremoveanentryforacollectionintheCurrenthierarchyintheagency.

1460-ERROR_CLUSTER_COULD_NOT_CREATE_DATABASE_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewdatabaseinthePlanhierarchyintheagency.

1461-ERROR_CLUSTER_COULD_NOT_CREATE_DATABASEWillberaisedwhenacoordinatorinaclusternoticesthatsomeDBServersreportproblemswhencreatingdatabasesforanewclusterwidedatabase.

1462-ERROR_CLUSTER_COULD_NOT_REMOVE_DATABASE_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanentryforadatabaseinthePlanhierarchyintheagency.

1463-ERROR_CLUSTER_COULD_NOT_REMOVE_DATABASE_IN_CURRENTWillberaisedwhenacoordinatorinaclustercannotremoveanentryforadatabaseintheCurrenthierarchyintheagency.

1464-ERROR_CLUSTER_SHARD_GONEWillberaisedwhenacoordinatorinaclustercannotdeterminetheshardthatisresponsibleforagivendocument.


707

1465-ERROR_CLUSTER_CONNECTION_LOSTWillberaisedwhenacoordinatorinaclusterlosesanHTTPconnectiontoaDBserverintheclusterwhilsttransferringdata.

1466-ERROR_CLUSTER_MUST_NOT_SPECIFY_KEYWillberaisedwhenacoordinatorinaclusterfindsthatthe_keyattributewasspecifiedinashardedcollectiontheusesnotonly_keyasshardingattribute.

1467-ERROR_CLUSTER_GOT_CONTRADICTING_ANSWERSWillberaisedifacoordinatorinaclustergetsconflictingresultsfromdifferentshards,whichshouldneverhappen.

1468-ERROR_CLUSTER_NOT_ALL_SHARDING_ATTRIBUTES_GIVENWillberaisedifacoordinatortriestofindoutwhichshardisresponsibleforapartialdocument,butcannotdothisbecausenotallshardingattributesarespecified.

1469-ERROR_CLUSTER_MUST_NOT_CHANGE_SHARDING_ATTRIBUTESWillberaisedifthereisanattempttoupdatethevalueofashardattribute.

1470-ERROR_CLUSTER_UNSUPPORTEDWillberaisedwhenthereisanattempttocarryoutanoperationthatisnotsupportedinthecontextofashardedcollection.

1471-ERROR_CLUSTER_ONLY_ON_COORDINATORWillberaisedifthereisanattempttorunacoordinator-onlyoperationonadifferenttypeofnode.

1472-ERROR_CLUSTER_READING_PLAN_AGENCYWillberaisedifacoordinatororDBservercannotreadthePlanintheagency.

1473-ERROR_CLUSTER_COULD_NOT_TRUNCATE_COLLECTIONWillberaisedifacoordinatorcannottruncateallshardsofaclustercollection.

1474-ERROR_CLUSTER_AQL_COMMUNICATIONWillberaisediftheinternalcommunicationoftheclusterforAQLproducesanerror.

1475-ERROR_ARANGO_DOCUMENT_NOT_FOUND_OR_SHARDING_ATTRIBUTES_CHANGEDWillberaisedwhenadocumentwithagivenidentifierorhandleisunknown,oriftheshardingattributeshavebeenchangedinaREPLACEoperationinthecluster.

1476-ERROR_CLUSTER_COULD_NOT_DETERMINE_IDWillberaisedifaclusterserveratstartupcouldnotdetermineitsownIDfromthelocalinfoprovided.

1477-ERROR_CLUSTER_ONLY_ON_DBSERVERWillberaisedifthereisanattempttorunaDBserver-onlyoperationonadifferenttypeofnode.

1478-ERROR_CLUSTER_BACKEND_UNAVAILABLEWillberaisedifarequireddbservercan'tbereached.

1479-ERROR_CLUSTER_UNKNOWN_CALLBACK_ENDPOINTAnendpointcouldn'tbefound

1480-ERROR_CLUSTER_AGENCY_STRUCTURE_INVALIDThestructureintheagencyisinvalid

1481-ERROR_CLUSTER_AQL_COLLECTION_OUT_OF_SYNCWillberaisedifacollectionneededduringqueryexecutionisoutofsync.Thiscurrentlycanonlyhappenwhenusingsatellitecollections

1482-ERROR_CLUSTER_COULD_NOT_CREATE_INDEX_IN_PLANWillberaisedwhenacoordinatorinaclustercannotcreateanentryforanewindexinthePlanhierarchyintheagency.

1483-ERROR_CLUSTER_COULD_NOT_DROP_INDEX_IN_PLANWillberaisedwhenacoordinatorinaclustercannotremoveanindexfromthePlanhierarchyintheagency.

1484-ERROR_CLUSTER_CHAIN_OF_DISTRIBUTESHARDSLIKEWillberaisedifonetriestocreateacollectionwithadistributeShardsLikeattributewhichpointstoanothercollectionthatalsohasone.


708

1485-ERROR_CLUSTER_MUST_NOT_DROP_COLL_OTHER_DISTRIBUTESHARDSLIKEWillberaisedifonetriestodropacollectiontowhichanothercollectionpointswithitsdistributeShardsLikeattribute.

1486-ERROR_CLUSTER_UNKNOWN_DISTRIBUTESHARDSLIKEWillberaisedifonetriestocreateacollectionwhichpointstoanunknowncollectioninitsdistributeShardsLikeattribute.

1487-ERROR_CLUSTER_INSUFFICIENT_DBSERVERSWillberaisedifonetriestocreateacollectionwithareplicationFactorgreaterthantheavailablenumberofDBServers.

1488-ERROR_CLUSTER_COULD_NOT_DROP_FOLLOWERWillberaisedifafollowerthatoughttobedroppedcouldnotbedroppedintheagency(underCurrent).

1489-ERROR_CLUSTER_SHARD_LEADER_REFUSES_REPLICATIONWillberaisedifareplicationoperationisrefusedbyashardleader.

1490-ERROR_CLUSTER_SHARD_FOLLOWER_REFUSES_OPERATIONWillberaisedifanon-replicationoperationisrefusedbyashardfollower.

1491-ERROR_CLUSTER_SHARD_LEADER_RESIGNEDWillberaisedifanon-replicationoperationisrefusedbyaformershardleaderthathasfoundoutthatitisnolongertheleader.

1492-ERROR_CLUSTER_AGENCY_COMMUNICATION_FAILEDWillberaisedifaftervariousretriesanagencyoperationcouldnotbeperformedsuccessfully.

1493-ERROR_CLUSTER_DISTRIBUTE_SHARDS_LIKE_REPLICATION_FACTORWillberaisedifintendedreplicationfactordoesnotmatchthatoftheprototypeshardgivenindistributeShardsLikeparameter.

1494-ERROR_CLUSTER_DISTRIBUTE_SHARDS_LIKE_NUMBER_OF_SHARDSWillberaisedifintendednumberofshardsdoesnotmatchthatoftheprototypeshardgivenindistributeShardsLikeparameter.

1495-ERROR_CLUSTER_LEADERSHIP_CHALLENGE_ONGOINGWillberaisedwhenserversarecurrentlycompetingforleadership,andtheresultisstillunknown.

1496-ERROR_CLUSTER_NOT_LEADERWillberaisedwhenanoperationissenttoanon-leadingserver.

ArangoDBqueryerrors

1500-ERROR_QUERY_KILLEDWillberaisedwhenarunningqueryiskilledbyanexplicitadmincommand.

1501-ERROR_QUERY_PARSEWillberaisedwhenqueryisparsedandisfoundtobesyntacticallyinvalid.

1502-ERROR_QUERY_EMPTYWillberaisedwhenanemptyqueryisspecified.

1503-ERROR_QUERY_SCRIPTWillberaisedwhenaruntimeerroriscausedbythequery.

1504-ERROR_QUERY_NUMBER_OUT_OF_RANGEWillberaisedwhenanumberisoutsidetheexpectedrange.

1510-ERROR_QUERY_VARIABLE_NAME_INVALIDWillberaisedwhenaninvalidvariablenameisused.

1511-ERROR_QUERY_VARIABLE_REDECLAREDWillberaisedwhenavariablegetsre-assignedinaquery.

1512-ERROR_QUERY_VARIABLE_NAME_UNKNOWNWillberaisedwhenanunknownvariableisusedorthevariableisundefinedthecontextitisused.

1521-ERROR_QUERY_COLLECTION_LOCK_FAILEDWillberaisedwhenareadlockonthecollectioncannotbeacquired.


709

1522-ERROR_QUERY_TOO_MANY_COLLECTIONSWillberaisedwhenthenumberofcollectionsinaqueryisbeyondtheallowedvalue.

1530-ERROR_QUERY_DOCUMENT_ATTRIBUTE_REDECLAREDWillberaisedwhenadocumentattributeisre-assigned.

1540-ERROR_QUERY_FUNCTION_NAME_UNKNOWNWillberaisedwhenanundefinedfunctioniscalled.

1541-ERROR_QUERY_FUNCTION_ARGUMENT_NUMBER_MISMATCHWillberaisedwhenthenumberofargumentsusedinafunctioncalldoesnotmatchtheexpectednumberofargumentsforthefunction.

1542-ERROR_QUERY_FUNCTION_ARGUMENT_TYPE_MISMATCHWillberaisedwhenthetypeofanargumentusedinafunctioncalldoesnotmatchtheexpectedargumenttype.

1543-ERROR_QUERY_INVALID_REGEXWillberaisedwhenaninvalidregexargumentvalueisusedinacalltoafunctionthatexpectsaregex.

1550-ERROR_QUERY_BIND_PARAMETERS_INVALIDWillberaisedwhenthestructureofbindparameterspassedhasanunexpectedformat.

1551-ERROR_QUERY_BIND_PARAMETER_MISSINGWillberaisedwhenabindparameterwasdeclaredinthequerybutthequeryisbeingexecutedwithnovalueforthatparameter.

1552-ERROR_QUERY_BIND_PARAMETER_UNDECLAREDWillberaisedwhenavaluegetsspecifiedforanundeclaredbindparameter.

1553-ERROR_QUERY_BIND_PARAMETER_TYPEWillberaisedwhenabindparameterhasaninvalidvalueortype.

1560-ERROR_QUERY_INVALID_LOGICAL_VALUEWillberaisedwhenanon-booleanvalueisusedinalogicaloperation.

1561-ERROR_QUERY_INVALID_ARITHMETIC_VALUEWillberaisedwhenanon-numericvalueisusedinanarithmeticoperation.

1562-ERROR_QUERY_DIVISION_BY_ZEROWillberaisedwhenthereisanattempttodividebyzero.

1563-ERROR_QUERY_ARRAY_EXPECTEDWillberaisedwhenanon-arrayoperandisusedforanoperationthatexpectsanarrayargumentoperand.

1569-ERROR_QUERY_FAIL_CALLEDWillberaisedwhenthefunctionFAIL()iscalledfrominsideaquery.

1570-ERROR_QUERY_GEO_INDEX_MISSINGWillberaisedwhenageorestrictionwasspecifiedbutnosuitablegeoindexisfoundtoresolveit.

1571-ERROR_QUERY_FULLTEXT_INDEX_MISSINGWillberaisedwhenafulltextqueryisperformedonacollectionwithoutasuitablefulltextindex.

1572-ERROR_QUERY_INVALID_DATE_VALUEWillberaisedwhenavaluecannotbeconvertedtoadate.

1573-ERROR_QUERY_MULTI_MODIFYWillberaisedwhenanAQLquerycontainsmorethanonedata-modifyingoperation.

1574-ERROR_QUERY_INVALID_AGGREGATE_EXPRESSIONWillberaisedwhenanAQLquerycontainsaninvalidaggregateexpression.

1575-ERROR_QUERY_COMPILE_TIME_OPTIONSWillberaisedwhenanAQLdata-modificationquerycontainsoptionsthatcannotbefiguredoutatquerycompiletime.

1576-ERROR_QUERY_EXCEPTION_OPTIONSWillberaisedwhenanAQLdata-modificationquerycontainsaninvalidoptionsspecification.


710

1577-ERROR_QUERY_COLLECTION_USED_IN_EXPRESSIONWillberaisedwhenacollectionisusedasanoperandinanAQLexpression.

1578-ERROR_QUERY_DISALLOWED_DYNAMIC_CALLWillberaisedwhenadynamicfunctioncallismadetoafunctionthatcannotbecalleddynamically.

1579-ERROR_QUERY_ACCESS_AFTER_MODIFICATIONWillberaisedwhencollectiondataareaccessedafteradata-modificationoperation.

AQLuserfunctionerrors1580-ERROR_QUERY_FUNCTION_INVALID_NAMEWillberaisedwhenauserfunctionwithaninvalidnameisregistered.

1581-ERROR_QUERY_FUNCTION_INVALID_CODEWillberaisedwhenauserfunctionisregisteredwithinvalidcode.

1582-ERROR_QUERY_FUNCTION_NOT_FOUNDWillberaisedwhenauserfunctionisaccessedbutnotfound.

1583-ERROR_QUERY_FUNCTION_RUNTIME_ERRORWillberaisedwhenauserfunctionthrowsaruntimeexception.

AQLqueryregistryerrors

1590-ERROR_QUERY_BAD_JSON_PLANWillberaisedwhenanHTTPAPIforaquerygotaninvalidJSONobject.

1591-ERROR_QUERY_NOT_FOUNDWillberaisedwhenanIdofaqueryisnotfoundbytheHTTPAPI.

1592-ERROR_QUERY_IN_USEWillberaisedwhenanIdofaqueryisfoundbytheHTTPAPIbutthequeryisinuse.

ArangoDBcursorerrors1600-ERROR_CURSOR_NOT_FOUNDWillberaisedwhenacursorisrequestedviaitsidbutacursorwiththatidcannotbefound.

1601-ERROR_CURSOR_BUSYWillberaisedwhenacursorisrequestedviaitsidbutaconcurrentrequestisstillusingthecursor.

ArangoDBtransactionerrors

1650-ERROR_TRANSACTION_INTERNALWillberaisedwhenawrongusageoftransactionsisdetected.thisisaninternalerrorandindicatesabuginArangoDB.

1651-ERROR_TRANSACTION_NESTEDWillberaisedwhentransactionsarenested.

1652-ERROR_TRANSACTION_UNREGISTERED_COLLECTIONWillberaisedwhenacollectionisusedinthemiddleofatransactionbutwasnotregisteredattransactionstart.

1653-ERROR_TRANSACTION_DISALLOWED_OPERATIONWillberaisedwhenadisallowedoperationiscarriedoutinatransaction.

1654-ERROR_TRANSACTION_ABORTEDWillberaisedwhenatransactionwasaborted.


711

Usermanagementerrors

1700-ERROR_USER_INVALID_NAMEWillberaisedwhenaninvalidusernameisused.

1701-ERROR_USER_INVALID_PASSWORDWillberaisedwhenaninvalidpasswordisused.

1702-ERROR_USER_DUPLICATEWillberaisedwhenausernamealreadyexists.

1703-ERROR_USER_NOT_FOUNDWillberaisedwhenausernameisupdatedthatdoesnotexist.

1704-ERROR_USER_CHANGE_PASSWORDWillberaisedwhentheusermustchangehispassword.

1705-ERROR_USER_EXTERNALWillberaisedwhentheuserisauthenicatedbyanexternalserver.

Servicemanagementerrors(legacy)

ThesehavebeensupercededbytheFoxxmanagementerrorsinpublicAPIs.

1750-ERROR_SERVICE_INVALID_NAMEWillberaisedwhenaninvalidservicenameisspecified.

1751-ERROR_SERVICE_INVALID_MOUNTWillberaisedwhenaninvalidmountisspecified.

1752-ERROR_SERVICE_DOWNLOAD_FAILEDWillberaisedwhenaservicedownloadfromthecentralrepositoryfailed.

1753-ERROR_SERVICE_UPLOAD_FAILEDWillberaisedwhenaserviceuploadfromtheclienttotheArangoDBserverfailed.

LDAPerrors

1800-ERROR_LDAP_CANNOT_INITcannotinitaLDAPconnection

1801-ERROR_LDAP_CANNOT_SET_OPTIONcannotsetaLDAPoption

1802-ERROR_LDAP_CANNOT_BINDcannotbindtoaLDAPserver

1803-ERROR_LDAP_CANNOT_UNBINDcannotunbindfromaLDAPserver

1804-ERROR_LDAP_CANNOT_SEARCHcannotsearchtheLDAPserver

1805-ERROR_LDAP_CANNOT_START_TLScannotstaraTLSLDAPsession

1806-ERROR_LDAP_FOUND_NO_OBJECTSLDAPdidn'tfoundanyobjectswiththespecifiedsearchquery

1807-ERROR_LDAP_NOT_ONE_USER_FOUNDLDAPfoundzerooremorethanoneuser


712

1808-ERROR_LDAP_USER_NOT_IDENTIFIEDLDAPfoundauser,butitsnotthedesiredone

1820-ERROR_LDAP_INVALID_MODEcantdistinguishavalidmodeforprovidedldapconfiguration

Taskerrors

1850-ERROR_TASK_INVALID_IDWillberaisedwhenataskiscreatedwithaninvalidid.

1851-ERROR_TASK_DUPLICATE_IDWillberaisedwhenataskidiscreatedwithaduplicateid.

1852-ERROR_TASK_NOT_FOUNDWillberaisedwhenataskwiththespecifiedidcouldnotbefound.

Graph/traversalerrors

1901-ERROR_GRAPH_INVALID_GRAPHWillberaisedwhenaninvalidnameispassedtotheserver.

1902-ERROR_GRAPH_COULD_NOT_CREATE_GRAPHWillberaisedwhenaninvalidname,verticesoredgesispassedtotheserver.

1903-ERROR_GRAPH_INVALID_VERTEXWillberaisedwhenaninvalidvertexidispassedtotheserver.

1904-ERROR_GRAPH_COULD_NOT_CREATE_VERTEXWillberaisedwhenthevertexcouldnotbecreated.

1905-ERROR_GRAPH_COULD_NOT_CHANGE_VERTEXWillberaisedwhenthevertexcouldnotbechanged.

1906-ERROR_GRAPH_INVALID_EDGEWillberaisedwhenaninvalidedgeidispassedtotheserver.

1907-ERROR_GRAPH_COULD_NOT_CREATE_EDGEWillberaisedwhentheedgecouldnotbecreated.

1908-ERROR_GRAPH_COULD_NOT_CHANGE_EDGEWillberaisedwhentheedgecouldnotbechanged.

1909-ERROR_GRAPH_TOO_MANY_ITERATIONSWillberaisedwhentoomanyiterationsaredoneinagraphtraversal.

1910-ERROR_GRAPH_INVALID_FILTER_RESULTWillberaisedwhenaninvalidfilterresultisreturnedinagraphtraversal.

1920-ERROR_GRAPH_COLLECTION_MULTI_USEanedgecollectionmayonlybeusedonceinoneedgedefinitionofagraph.,

1921-ERROR_GRAPH_COLLECTION_USE_IN_MULTI_GRAPHSisalreadyusedbyanothergraphinadifferentedgedefinition.,

1922-ERROR_GRAPH_CREATE_MISSING_NAMEagraphnameisrequiredtocreateagraph.,

1923-ERROR_GRAPH_CREATE_MALFORMED_EDGE_DEFINITIONtheedgedefinitionismalformed.Ithastobeanarrayofobjects.,

1924-ERROR_GRAPH_NOT_FOUNDagraphwiththisnamecouldnotbefound.,


713

1925-ERROR_GRAPH_DUPLICATEagraphwiththisnamealreadyexists.,

1926-ERROR_GRAPH_VERTEX_COL_DOES_NOT_EXISTthespecifiedvertexcollectiondoesnotexistorisnotpartofthegraph.,

1927-ERROR_GRAPH_WRONG_COLLECTION_TYPE_VERTEXthecollectionisnotavertexcollection.,

1928-ERROR_GRAPH_NOT_IN_ORPHAN_COLLECTIONVertexcollectionnotinorphancollectionofthegraph.,

1929-ERROR_GRAPH_COLLECTION_USED_IN_EDGE_DEFThecollectionisalreadyusedinanedgedefinitionofthegraph.,

1930-ERROR_GRAPH_EDGE_COLLECTION_NOT_USEDTheedgecollectionisnotusedinanyedgedefinitionofthegraph.,

1932-ERROR_GRAPH_NO_GRAPH_COLLECTIONcollection_graphsdoesnotexist.,

1933-ERROR_GRAPH_INVALID_EXAMPLE_ARRAY_OBJECT_STRINGInvalidexampletype.HastobeString,ArrayorObject.,

1934-ERROR_GRAPH_INVALID_EXAMPLE_ARRAY_OBJECTInvalidexampletype.HastobeArrayorObject.,

1935-ERROR_GRAPH_INVALID_NUMBER_OF_ARGUMENTSInvalidnumberofarguments.Expected:,

1936-ERROR_GRAPH_INVALID_PARAMETERInvalidparametertype.,

1937-ERROR_GRAPH_INVALID_IDInvalidid,

1938-ERROR_GRAPH_COLLECTION_USED_IN_ORPHANSThecollectionisalreadyusedintheorphansofthegraph.,

1939-ERROR_GRAPH_EDGE_COL_DOES_NOT_EXISTthespecifiededgecollectiondoesnotexistorisnotpartofthegraph.,

1940-ERROR_GRAPH_EMPTYTherequestedgraphhasnoedgecollections.

Sessionerrors1950-ERROR_SESSION_UNKNOWNWillberaisedwhenaninvalid/unknownsessionidispassedtotheserver.

1951-ERROR_SESSION_EXPIREDWillberaisedwhenasessionisexpired.

SimpleClienterrors

2000-SIMPLE_CLIENT_UNKNOWN_ERRORThiserrorshouldnothappen.

2001-SIMPLE_CLIENT_COULD_NOT_CONNECTWillberaisedwhentheclientcouldnotconnecttotheserver.

2002-SIMPLE_CLIENT_COULD_NOT_WRITEWillberaisedwhentheclientcouldnotwritedata.


714

2003-SIMPLE_CLIENT_COULD_NOT_READWillberaisedwhentheclientcouldnotreaddata.

Communicatorerrors

2100-COMMUNICATOR_REQUEST_ABORTEDRequestwasaborted.

2101-COMMUNICATOR_DISABLEDCommunicationwasdisabled.

Foxxmanagementerrors

3000-ERROR_MALFORMED_MANIFEST_FILETheservicemanifestfileisnotwell-formedJSON.

3001-ERROR_INVALID_SERVICE_MANIFESTTheservicemanifestcontainsinvalidvalues.

3002-ERROR_SERVICE_FILES_MISSINGTheservicefolderorbundledoesnotexistonthisserver.

3003-ERROR_SERVICE_FILES_OUTDATEDThelocalservicebundledoesnotmatchthechecksuminthedatabase.

3004-ERROR_INVALID_FOXX_OPTIONSTheserviceoptionscontaininvalidvalues.

3007-ERROR_INVALID_MOUNTPOINTTheservicemountpathcontainsinvalidcharacters.

3009-ERROR_SERVICE_NOT_FOUNDNoservicefoundatthegivenmountpath.

3010-ERROR_SERVICE_NEEDS_CONFIGURATIONTheserviceismissingconfigurationordependencies.

3011-ERROR_SERVICE_MOUNTPOINT_CONFLICTAservicealreadyexistsatthegivenmountpath.

3012-ERROR_SERVICE_MANIFEST_NOT_FOUNDTheservicedirectorydoesnotcontainamanifestfile.

3013-ERROR_SERVICE_OPTIONS_MALFORMEDTheserviceoptionsarenotwell-formedJSON.

3014-ERROR_SERVICE_SOURCE_NOT_FOUNDThesourcepathdoesnotmatchafileordirectory.

3015-ERROR_SERVICE_SOURCE_ERRORThesourcepathcouldnotberesolved.

3016-ERROR_SERVICE_UNKNOWN_SCRIPTTheservicedoesnothaveascriptwiththisname.

JavaScriptmoduleloadererrors3100-ERROR_MODULE_NOT_FOUNDThemodulepathcouldnotberesolved.


715

3101-ERROR_MODULE_SYNTAX_ERRORThemodulecouldnotbeparsedbecauseofasyntaxerror.

3103-ERROR_MODULE_FAILUREFailedtoinvokethemoduleinitscontext.

Enterpriseerrors

4000-ERROR_NO_SMART_COLLECTIONTherequestedcollectionneedstobesmart,butitain't

4001-ERROR_NO_SMART_GRAPH_ATTRIBUTEThegivendocumentdoesnothavethesmartgraphattributeset.

4002-ERROR_CANNOT_DROP_SMART_COLLECTIONThissmartcollectioncannotbedropped,itdictatesshardinginthegraph.

4003-ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTEInasmartvertexcollection_keymustbeprefixedwiththevalueofthesmartgraphattribute.

4004-ERROR_ILLEGAL_SMART_GRAPH_ATTRIBUTEThegivensmartGraphattributeisillegalandconnotbeusedforsharding.Allsystemattributesareforbidden.

Clusterrepairerrors

5000-ERROR_CLUSTER_REPAIRS_FAILEDGeneralerrorduringclusterrepairs

5001-ERROR_CLUSTER_REPAIRS_NOT_ENOUGH_HEALTHYWillberaisedwhen,duringrepairDistributeShardsLike,theremustbeafreedbservertomoveashard,butthereisnocandidateornoneishealthy.

5002-ERROR_CLUSTER_REPAIRS_REPLICATION_FACTOR_VIOLATEDWillberaisedonvariousinconsistenciesregardingthereplicationfactor

5003-ERROR_CLUSTER_REPAIRS_NO_DBSERVERSWillberaisedifacollectionthatisfixedhassomeshardwithoutDBServers

5004-ERROR_CLUSTER_REPAIRS_MISMATCHING_LEADERSWillberaisedifashardincollectionanditsprototypeinthecorrespondingdistributeShardsLikecollectionhavemismatchingleaders(whentheyshouldalreadyhavebeenfixed)

5005-ERROR_CLUSTER_REPAIRS_MISMATCHING_FOLLOWERSWillberaisedifashardincollectionanditsprototypeinthecorrespondingdistributeShardsLikecollectiondon'thavethesamefollowers(whentheyshouldalreadyhavebeenadjusted)

5006-ERROR_CLUSTER_REPAIRS_INCONSISTENT_ATTRIBUTESWillberaisedifacollectionthatisfixeddoes(not)havedistributeShardsLikewhenitisexpected,ordoes(not)haverepairingDistributeShardsLikewhenitisexpected

5007-ERROR_CLUSTER_REPAIRS_MISMATCHING_SHARDSWillberaisedifinacollectionanditsdistributeShardsLikeprototypecollectionsomeshardanditsprototypehaveanunequalnumberofDBServers

5008-ERROR_CLUSTER_REPAIRS_JOB_FAILEDWillberaisedifamoveshardjobintheagencyfailedduringclusterrepairs

5009-ERROR_CLUSTER_REPAIRS_JOB_DISAPPEAREDWillberaisedifamoveshardjobintheagencycannotbefoundanymorebeforeitfinished

5010-ERROR_CLUSTER_REPAIRS_OPERATION_FAILEDWillberaisedifanagencytransactionfailedduringeithersendingorexecutingit.


716

Agencyerrors

20001-ERROR_AGENCY_INQUIRY_SYNTAXInquiryhandlesalistofstringclientIds:[,...].

20011-ERROR_AGENCY_INFORM_MUST_BE_OBJECTTheinformmessageintheagencymustbeanobject.

20012-ERROR_AGENCY_INFORM_MUST_CONTAIN_TERMTheinformmessageintheagencymustcontainauintparameter'term'.

20013-ERROR_AGENCY_INFORM_MUST_CONTAIN_IDTheinformmessageintheagencymustcontainastringparameter'id'.

20014-ERROR_AGENCY_INFORM_MUST_CONTAIN_ACTIVETheinformmessageintheagencymustcontainanarray'active'.

20015-ERROR_AGENCY_INFORM_MUST_CONTAIN_POOLTheinformmessageintheagencymustcontainanobject'pool'.

20016-ERROR_AGENCY_INFORM_MUST_CONTAIN_MIN_PINGTheinformmessageintheagencymustcontainanobject'minping'.

20017-ERROR_AGENCY_INFORM_MUST_CONTAIN_MAX_PINGTheinformmessageintheagencymustcontainanobject'maxping'.

20018-ERROR_AGENCY_INFORM_MUST_CONTAIN_TIMEOUT_MULTTheinformmessageintheagencymustcontainanobject'timeoutMult'.

20020-ERROR_AGENCY_INQUIRE_CLIENT_ID_MUST_BE_STRINGInquirybyclientIdfailed

20021-ERROR_AGENCY_CANNOT_REBUILD_DBSWillberaisedifthereadDBorthespearHeadcannotberebuiltfromthereplicatedlog.

Supervisionerrors

20501-ERROR_SUPERVISION_GENERAL_FAILUREGeneralsupervisionfailure.

Dispatchererrors

21001-ERROR_DISPATCHER_IS_STOPPINGWillbereturnedifashutdownisinprogress.

21002-ERROR_QUEUE_UNKNOWNWillbereturnedifaqueuewiththisnamedoesnotexist.

21003-ERROR_QUEUE_FULLWillbereturnedifaqueuewiththisnameisfull.


717

Glossary

Collection

Acollectionconsistsofdocuments.Itisuniquelyidentifiedbyitscollectionidentifier.Italsohasauniquenamethatclientsshouldusetoidentifyandaccessit.Collectionscanberenamed.Itwillchangethecollectionname,butnotthecollectionidentifier.Collectionscontaindocumentsofaspecifictype.Therearecurrentlytwotypes:document(default)andedge.Thetypeisspecifiedbytheuserwhenthecollectioniscreated,andcannotbechangedlater.

CollectionIdentifier

Acollectionidentifieridentifiesacollectioninadatabase.Itisastringvalueandisuniquewithinthedatabase.UptoincludingArangoDB1.1,thecollectionidentifierhasbeenaclient'sprimarymeanstoaccesscollections.StartingwithArangoDB1.2,clientsshouldinsteaduseacollection'suniquenametoaccessacollectioninsteadofitsidentifier.

ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintaincollectionidsinternally.Whenreturningcollectionidstoclients,ArangoDBwillputthemintoastringtoensurethecollectionidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreatthecollectionidsreturnedbyArangoDBasopaquestringswhentheystoreoruseitlocally.

CollectionName

Acollectionnameidentifiesacollectioninadatabase.Itisastringandisuniquewithinthedatabase.Unlikethecollectionidentifieritissuppliedbythecreatorofthecollection.Thecollectionnamemustconsistofletters,digits,andthe_(underscore)and-(dash)charactersonly.PleaserefertoNamingConventionsformoreinformationonvalidcollectionnames.

Database

ArangoDBcanhandlemultipledatabasesinthesameserverinstance.Databasescanbeusedtologicallygroupandseparatedata.AnArangoDBdatabaseconsistsofcollectionsanddedicateddatabase-specificworkerprocesses.

Adatabasecontainsitsowncollections(whichcannotbeaccessedfromotherdatabases),Foxxapplications,andreplicationloggersandappliers.EachArangoDBdatabasecontainsitsownsystemcollections(e.g._users,_replication,...).

TherewillalwaysbeatleastonedatabaseinArangoDB.Thisisthedefaultdatabase,named_system.Thisdatabasecannotbedropped,andprovidesspecialoperationsforcreating,dropping,andenumeratingdatabases.Userscancreateadditionaldatabasesandgivethemuniquenamestoaccessthemlater.Databasemanagementoperationscannotbeinitiatedfromoutofuser-defineddatabases.

WhenArangoDBisaccessedviaitsHTTPRESTAPI,thedatabasenameisreadfromthefirstpartoftherequestURIpath(e.g./_db/_system/...).IftherequestURIdoesnotcontainadatabasename,thedatabasenameisautomaticallyderivedfromtheendpoint.PleaserefertoDatabaseEndpointformoreinformation.

DatabaseName

AsingleArangoDBinstancecanhandlemultipledatabasesinparallel.Whenmultipledatabasesareused,eachdatabasemustbegivenauniquename.Thisnameisusedtouniquelyidentifyadatabase.ThedefaultdatabaseinArangoDBisnamed_system.

Thedatabasenameisastringconsistingofonlyletters,digitsandthe_(underscore)and-(dash)characters.User-defineddatabasenamesmustalwaysstartwithaletter.Databasenamesiscase-sensitive.

DatabaseOrganization

AsingleArangoDBinstancecanhandlemultipledatabasesinparallel.Bydefault,therewillbeatleastonedatabase,whichisnamed_system.

Databasesarephysicallystoredinseparatesub-directoriesunderneaththedatabasedirectory,whichitselfresidesintheinstance'sdatadirectory.

Glossary

718

Eachdatabasehasitsownsub-directory,nameddatabase-.Thedatabasedirectorycontainssub-directoriesforthecollectionsofthedatabase,andafilenamedparameter.json.Thisfilecontainsthedatabaseidandname.

InanexampleArangoDBinstancewhichhastwodatabases,thefilesystemlayoutcouldlooklikethis:

data/#theinstance'sdatadirectory

databases/#sub-directorycontainingalldatabases'data

database-<id>/#sub-directoryforasingledatabase

parameter.json#filecontainingdatabaseidandname

collection-<id>/#directorycontainingdataaboutacollection

database-<id>/#sub-directoryforanotherdatabase

parameter.json#filecontainingdatabaseidandname



Foxxapplicationsarealsoorganizedindatabase-specificdirectoriesinsidetheapplicationpath.Thefilesystemlayoutcouldlooklikethis:

apps/#theinstance'sapplicationdirectory

system/#systemapplications(canbeignored)

_db/#sub-directorycontainingdatabase-specificapplications

<database-name>/#sub-directoryforasingledatabase

<mountpoint>/APP#sub-directoryforasingleapplication


<database-name>/#sub-directoryforanotherdatabase


Document

DocumentsinArangoDBareJSONobjects.Theseobjectscanbenested(toanydepth)andmaycontainarrays.Eachdocumentisuniquelyidentifiedbyitsdocumenthandle.

DocumentEtag

Thedocumentrevision(_revvalue)enclosedindoublequotes.TherevisionisreturnedbyseveralHTTPAPImethodsintheEtagHTTPheader.

DocumentHandle

Adocumenthandleuniquelyidentifiesadocumentinthedatabase.Itisastringandconsistsofthecollection'snameandthedocumentkey(_keyattribute)separatedby/.Thedocumenthandleisstoredinadocument's_idattribute.

DocumentKey

Adocumentkeyisastringthatuniquelyidentifiesadocumentinagivencollection.Itcanandshouldbeusedbyclientswhenspecificdocumentsaresearched.Documentkeysarestoredinthe_keyattributeofdocuments.ThekeyvaluesareautomaticallyindexedbyArangoDBinacollection'sprimaryindex.Thuslookingupadocumentbyitskeyisregularlyafastoperation.The_keyvalueofadocumentisimmutableoncethedocumenthasbeencreated.

Bydefault,ArangoDBwillauto-generateadocumentkeyifno_keyattributeisspecified,andusetheuser-specified_keyvalueotherwise.

Thisbehaviorcanbechangedonaper-collectionlevelbycreatingcollectionswiththekeyOptionsattribute.

UsingkeyOptionsitispossibletodisallowuser-specifiedkeyscompletely,ortoforceaspecificregimeforauto-generatingthe_keyvalues.

Therearesomerestrictionsforuser-definedkeys(seeNamingConventionsfordocumentkeys).

DocumentRevision

Glossary

719

AsArangoDBsupportsMVCC,documentscanexistinmorethanonerevision.ThedocumentrevisionistheMVCCtokenusedtoidentifyaparticularrevisionofadocument.Itisastringvaluecurrentlycontaininganintegernumberandisuniquewithinthelistofdocumentrevisionsforasingledocument.Documentrevisionscanbeusedtoconditionallyupdate,replaceordeletedocumentsinthedatabase.Inordertofindaparticularrevisionofadocument,youneedthedocumenthandleandthedocumentrevision.

Thedocumentrevisionisstoredinthe_revattributeofadocument,andissetandupdatedbyArangoDBautomatically.The_revvaluecannotbesetfromtheoutside.

ArangoDBcurrentlyuses64bitunsignedintegervaluestomaintaindocumentrevisionsinternally.Whenreturningdocumentrevisionstoclients,ArangoDBwillputthemintoastringtoensuretherevisionidisnotclippedbyclientsthatdonotsupportbigintegers.ClientsshouldtreattherevisionidreturnedbyArangoDBasanopaquestringwhentheystoreoruseitlocally.ThiswillallowArangoDBtochangetheformatofrevisionidslaterifthisshouldberequired.Clientscanuserevisionsidstoperformsimpleequality/non-equalitycomparisons(e.g.tocheckwhetheradocumenthaschangedornot),buttheyshouldnotuserevisionidstoperformgreater/lessthancomparisonswiththemtocheckifadocumentrevisionisolderthanoneanother,evenifthismightworkforsomecases.

Edge

Edgesarespecialdocumentsusedforconnectingotherdocumentsintoagraph.Anedgedescribestheconnectionbetweentwodocumentsusingtheinternalattributes:_fromand_to.Thesecontaindocumenthandles,namelythestart-pointandtheend-pointoftheedge.

EdgeCollection

Edgecollectionsarecollectionsthatstoreedges.

EdgeDefinition

Edgedefinitionsarepartsofthedefinitionofnamedgraphs.Theydescribewhichedgecollectionsconnectwhichvertexcollections.

GeneralGraph

Modulemaintaininggraphsetupinthe_graphscollection-akanamedgraphs.Configureswhichedgecollectionsrelatetowhichvertexcollections.Ensuresgraphconsistencyinmodificationqueries.

NamedGraphs

Namedgraphsenforceconsistencybetweenedgecollectionsandvertexcollections,soifyouremoveavertex,edgespointingtoitwillberemovedtoo.

Index

Indexesareusedtoallowfastaccesstodocumentsinacollection.Allcollectionshaveaprimaryindex,whichisthedocument's_keyattribute.Thisindexcannotbedroppedorchanged.

Edgecollectionswillalsohaveanautomaticallycreatededgesindex,whichcannotbemodified.Thisindexprovidesquickaccesstodocumentsviathe_fromand_toattributes.

Mostuser-landindexescanbecreatedbydefiningthenamesoftheattributeswhichshouldbeindexed.Someindextypesallowindexingjustoneattribute(e.g.fulltextindex)whereasotherindextypesallowindexingmultipleattributes.

Indexingthesystemattribute_idinuser-definedindexesisnotsupportedbyanyindextype.

EdgesIndex

Anedgesindexisautomaticallycreatedforedgecollections.Itcontainsconnectionsbetweenvertexdocumentsandisinvokedwhentheconnectingedgesofavertexarequeried.Thereisnowaytoexplicitlycreateordeleteedgesindexes.

FulltextIndex

Glossary

720

Afulltextindexcanbeusedtofindwords,orprefixesofwordsinsidedocuments.Afulltextindexcanbedefinedononeattributeonly,andwillincludeallwordscontainedindocumentsthathaveatextualvalueintheindexattribute.SinceArangoDB2.6theindexwillalsoincludewordsfromtheindexattributeiftheindexattributeisanarrayofstrings,oranobjectwithstringvaluemembers.

Forexample,givenafulltextindexonthetranslationsattributeandthefollowingdocuments,thensearchingforлисаusingthefulltextindexwouldreturnonlythefirstdocument.SearchingfortheindexfortheexactstringFoxwouldreturnthefirsttwodocuments,andsearchingforprefix:Foxwouldreturnallthreedocuments:

{translations:{en:"fox",de:"Fuchs",fr:"renard",ru:"лиса"}}

{translations:"FoxistheEnglishtranslationoftheGermanwordFuchs"}

{translations:["ArangoDB","document","database","Foxx"]}

Iftheindexattributeisneitherastring,anobjectoranarray,itscontentswillnotbeindexed.Whenindexingthecontentsofanarrayattribute,anarraymemberwillonlybeincludedintheindexifitisastring.Whenindexingthecontentsofanobjectattribute,anobjectmembervaluewillonlybeincludedintheindexifitisastring.Otherdatatypesareignoredandnotindexed.

Onlywordswitha(specifiable)minimumlengthareindexed.Wordtokenizationisdoneusingthewordboundaryanalysisprovidedbylibicu,whichistakingintoaccounttheselectedlanguageprovidedatserverstart.Wordsareindexedintheirlower-casedform.Theindexsupportscompletematchqueries(fullwords)andprefixqueries.

GeoIndex

Ageoindexisusedtofindplacesonthesurfaceoftheearthfast.

IndexHandle

Anindexhandleuniquelyidentifiesanindexinthedatabase.Itisastringandconsistsofacollectionnameandanindexidentifierseparatedby/.

HashIndex

Ahashindexisusedtofinddocumentsbasedonexamples.Ahashindexcanbecreatedforoneormultipledocumentattributes.

Ahashindexwillonlybeusedbyqueriesifallindexedattributesarepresentintheexampleorsearchquery,andifallattributesarecomparedusingtheequality(==operator).Thatmeansthehashindexdoesnotsupportrangequeries.

AuniquehashindexhasanamortizedcomplexityofO(1)forlookup,insert,update,andremoveoperations.Thenon-uniquehashindexissimilar,butamortizedlookupperformanceisO(n),withnbeingthenumberofindexentrieswiththesamelookupvalue.

SkiplistIndex

Askiplistisasortedindextypethatcanbeusedtofindrangesofdocuments.

AnonymousGraphs

Youmayuseedgecollectionswithvertexcollectionswithoutthegraphmanagementfacilities.However,graphconsistencyisnotenforcedbythese.Ifyouremovevertices,youhavetoensurebyyourselvesedgespointingtothisvertexareremoved.Anonymousgraphsmaynotbebrowsedusinggraphviewerinthewebinterface.Thismaybefasterinsomescenarios.

Glossary

721