jumpstart: monitoring and tuning marklogic · 2018. 9. 29. · primarily gui -based but can export...
TRANSCRIPT
![Page 1: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/1.jpg)
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Silvano Ravotto and Marc Chiarini, MarkLogic Performance Engineering
JUMPSTART: MONITORING AND TUNING MARKLOGIC
![Page 2: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/2.jpg)
SLIDE: 2
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Agenda • Introduction
• Methodologies
• Monitoring
• Tools overview
• Platform specific tools
• Tuning
![Page 3: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/3.jpg)
INTRODUCTION
![Page 4: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/4.jpg)
SLIDE: 4
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Going Back in Time
![Page 5: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/5.jpg)
SLIDE: 5
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Picture = 34,140 words
![Page 6: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/6.jpg)
SLIDE: 6
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Application
Looks Familiar?
OS
Virtualization
CPU Disk Memory Network
Third Party Software
![Page 7: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/7.jpg)
SLIDE: 7
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
“Or se' tu quel Virgilio?”
http://www.brendangregg.com/
Great resource on the world of systems performance
“Systems Performance: Enterprise and the Cloud” (Book)
![Page 8: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/8.jpg)
METHODOLOGIES
![Page 9: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/9.jpg)
SLIDE: 9
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Methodologies
Procedures to analyze system or application performance
Provide a starting point
Different methodologies are suited for solving different classes of issues
Try more than one before accomplishing your goal
![Page 10: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/10.jpg)
SLIDE: 10
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Some Good Methodologies Whys Method
Drill-Down Analysis Method
USE Method ‒ For every resource, check:
Utilization
Saturation
Errors
![Page 11: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/11.jpg)
OBSERVABILITY TOOLS
![Page 12: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/12.jpg)
SLIDE: 12
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Observability Tools
In order to characterize performance, you need to observe it.
(Trust Nothing)
![Page 13: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/13.jpg)
SLIDE: 13
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Observability Tools Application Tools
MarkLogic Monitoring History
Infrastructure Tools
Amazon CloudWatch, Performance Co-Pilot, Nagios, NewRelic
OS Tools
Linux, Windows
![Page 14: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/14.jpg)
SLIDE: 14
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
MarkLogic Monitoring History (Application)
![Page 15: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/15.jpg)
SLIDE: 15
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
MarkLogic Monitoring History
MarkLogic server view
Covers several resources
Disk, CPU, Memory, Network, Server, Database
Can’t be used alone
MarkLogic process overloaded => incomplete picture
![Page 16: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/16.jpg)
SLIDE: 16
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Amazon CloudWatch (Infrastructure)
![Page 17: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/17.jpg)
SLIDE: 17
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Amazon CloudWatch
Collects and tracks metrics and log files
Set alarms
Monitors AWS resources
Amazon EC2 instances, EBS, ELB
Supports custom metrics generated by applications and services
![Page 18: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/18.jpg)
SLIDE: 18
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Performance Co-Pilot (Infrastructure)
PCP is a framework for system-level performance analysis
Collection, monitoring, and analysis of system metrics
Available on Linux, Windows, Mac OS
Easily extendable and flexible
MarkLogic PMDA - https://github.com/sravotto/marklogic-pcp-pmda
![Page 19: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/19.jpg)
SLIDE: 19
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Sample PCP Commands pmstat
High level metrics (like vmstat)
pminfo
List all the known metrics
pmdumptext
Dump performance metrics in ASCII table
![Page 20: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/20.jpg)
SLIDE: 20
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Sample PCP Graphical Tool
![Page 21: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/21.jpg)
Linux Tools
![Page 22: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/22.jpg)
SLIDE: 22
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Linux Performance Observability Tools
![Page 23: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/23.jpg)
SLIDE: 23
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Perf (Linux Kernel 2.6+) perf top
Generates and displays a performance counter profile in real time
perf record -F 99 -ag -- sleep 60
Record overall activity for 60 seconds
perf report --stdio
Display the activity
![Page 24: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/24.jpg)
SLIDE: 24
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Perf Report Example 80.54% 0.23% MarkLogic MarkLogic [.] xdmp::LetClause::flworEval | ---xdmp::LetClause::flworEval | |--95.87%-- xdmp::LetClause::flworEval | | | |--97.76%-- xdmp::LetClause::flworEval | | | | | |--99.79%-- xdmp::LetClause::flworEval | | | | | | | |--93.91%-- xdmp::FLWORExpr::eval | | | | xdmp::IfExpr::eval | | | | xdmp::EvalFLWOREnv::evalReturn | | | | xdmp::LetClause::flworEval | | | | xdmp::FLWORExpr::eval | | | | xdmp::FunctionCall::eval | | | | xdmp::WithNamespacesExpr::eval
![Page 25: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/25.jpg)
SLIDE: 25
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Sample Flame Graph (Ingestion) Saving
In-Memory Stand
Merges
Parsing + Indexing
![Page 26: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/26.jpg)
CASE STUDY
![Page 27: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/27.jpg)
SLIDE: 27
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Customer Case Study Main Use Cases
Raw ingest
Transformations of content
Extraction
Problem Statement
Sharp performance degradation of ingestion with increasing database size
SLA not longer met
![Page 28: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/28.jpg)
# sar –u
12:00:01 AM CPU %usr %nice %sys %iowait %steal %irq %soft %guest %idle
04:33:01 PM all 8.01 0.06 3.00 31.91 0.00 0.00 0.20 0.00 56.83
04:34:01 PM all 4.40 0.06 1.83 44.48 0.00 0.00 0.14 0.00 49.10
04:35:01 PM all 4.68 0.05 1.89 39.08 0.00 0.00 0.15 0.00 54.15
CPU Metrics
CPU is mostly waiting for I/O. Why?
![Page 29: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/29.jpg)
# sar -d
12:00:01 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
04:34:01 PM dev8-0 4984.60 120171.70 7192.76 25.55 117.69 23.61 0.21 104.57
04:35:01 PM dev8-0 4215.76 126061.58 6838.80 31.52 96.93 22.99 0.22 91.28
04:36:01 PM dev8-0 4126.01 135221.99 6944.88 34.46 95.97 23.28 0.22 90.15
04:37:02 PM dev8-0 4135.01 137442.28 7359.64 35.02 96.92 23.44 0.22 89.76
I/O Metrics
Reads are dominating writes during ingest. Why?
![Page 30: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/30.jpg)
Thread A
pread64()
svc::StandardRandomReadResult (…)
svc::StandardRandomReadTask::run()
svc::PooledThread::run()
Execution Stack Analysis
Execution stack (pstack) shows threads waiting on random reads.
Threads waiting on I/O originate from cts:uris. Why?
Thread B
sem_wait,svc::Semaphore::wait(bool)
svc::StandardRandomReadResultFuture::_finish()
…
xdmp::cts_uris(…)
![Page 31: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/31.jpg)
let $uri := cts:uris((), (), cts:element-value-query(xs:Qname(“ID”), $id))
return
if ($uri)
then update (...)
else insert (...)
Code Inspection
We found cts:uris calls had an element value query with a document ID.
Why is this impacting performance?
![Page 32: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/32.jpg)
SLIDE: 32
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
ID Lookup Using the Universal Index
HASH(ID) =A34E0
ID
List Index (One for each Stand)
A34DE-A34EA A34E0 ?
MEMORY
List Data (One for each Stand)
DISK (Term List)
Worst Case Disk Reads (Node) = Ingestion Rate * Number of Stands
![Page 33: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/33.jpg)
SLIDE: 33
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Tuning Advice
Avoid I/O to get the term list
With 20 stands/node and 500 DPS = 10,000 read operations per second!
Leverage a range index for these unique IDS
All the values are in memory
Change cts:element-value-query to cts:element-range-query
![Page 34: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/34.jpg)
SLIDE: 34
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Case Study Recap
Analyze each layer
Ask yourself why you see a particular behavior
Collect additional metrics as you drill down
Are the metrics telling you a different story?
Make changes once you understand the issue
Only one change at a time
![Page 35: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/35.jpg)
WINDOWS TOOLS
![Page 36: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/36.jpg)
SLIDE: 36
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Windows Performance Analysis Tools Primarily GUI-based but can export important data
Progressively more powerful and detailed:
Procexp (Process Explorer) – task manager
Procmon (Process Monitor) – record system calls and thread activity
WPA (Windows Performance Analyzer) – fine-grain detail on CPU, memory, disk, network activity, etc
![Page 37: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/37.jpg)
SLIDE: 37
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Procexp (Process Explorer) Comprehensive replacement for Windows Task Manager
Gives a sense of performance / problems in the moment.
Available at https://technet.microsoft.com/en-us/sysinternals/processexplorer.
![Page 38: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/38.jpg)
SLIDE: 38
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Procexp Main Window
![Page 39: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/39.jpg)
SLIDE: 39
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
MarkLogic Resources
![Page 40: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/40.jpg)
SLIDE: 40
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Performance Stats
![Page 41: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/41.jpg)
SLIDE: 41
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Disk & Network Stats
![Page 42: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/42.jpg)
SLIDE: 42
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Detailed Resources
![Page 43: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/43.jpg)
SLIDE: 43
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
![Page 44: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/44.jpg)
SLIDE: 44
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
![Page 45: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/45.jpg)
SLIDE: 45
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Procmon (Process Monitor) Records and filters Windows system call statistics
Can also do thread profiling
Useful for finding abnormal conditions/errors that can cause performance problems
Available at https://technet.microsoft.com/en-us/sysinternals/processmonitor
![Page 46: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/46.jpg)
SLIDE: 46
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Procmon Main Window
Start / Stop Capture
Start / Stop Autoscroll
Clear Log
Show Filesystem
Events
Show Proc/Thread
Events
![Page 47: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/47.jpg)
SLIDE: 47
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Resources Used by Visible Events
![Page 48: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/48.jpg)
SLIDE: 48
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Occurrences of Column Values
![Page 49: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/49.jpg)
SLIDE: 49
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Filesystem Activity Summary
![Page 50: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/50.jpg)
SLIDE: 50
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Procmon Workflow Capture a short trace
Determine event classes of interest
Capture a longer trace
Filter out disinteresting events such as successful system calls
Search for unusually high or low event counts
Search for unusual errors
Send stack traces of unexpected activity to ML support
![Page 51: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/51.jpg)
SLIDE: 51
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Procmon Workflow
![Page 52: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/52.jpg)
SLIDE: 52
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
WPA (Windows Performance Analyzer) Comprehensive, detailed tracing of all activities
On-screen highlighting and correlation of concurrent events
Available as part of Windows ADK at http://go.microsoft.com/fwlink/p/?LinkId=526740
Also be sure to check out Bruce Dawson’s UIforETW available at: https://randomascii.wordpress.com/2015/09/24/etw-central/
![Page 53: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/53.jpg)
SLIDE: 53
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
WPA Main Window
![Page 54: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/54.jpg)
SLIDE: 54
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Windows Tools Summary Similar to Linux there is a set of Windows tools that can be used to observe and
analyze MarkLogic performance at any level
Procexp is good for instantaneous observation
Procmon is good for finding unexpected behaviors in Windows system calls
WPA is good for correlating resource usage across MarkLogic functions
![Page 55: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/55.jpg)
TUNING
![Page 56: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/56.jpg)
SLIDE: 56
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Tuning Targets
Target Tunables
Application Queries
MarkLogic Server Indexing, Cache, MVCC settings
Operating System Huge Pages, Virtual Memory settings
Storage RAID Level, I/O Schedulers
![Page 57: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/57.jpg)
SLIDE: 57
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Tuning – Application Target Queries: Query Meters, Query Plan
Code: Profiler API
For more details, see the “Performance Tuning for Developers” session
![Page 58: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/58.jpg)
SLIDE: 58
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Tuning – Server Target Caches
Bigger caches → Less I/O
More partitions → May reduce contention
Indexes
Trade-off: Disk space vs performance
Multi-Version Concurrency Control (MVCC)
Contemporaneous vs nonblocking
![Page 59: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/59.jpg)
SLIDE: 59
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Tuning – OS Target Huge Pages
Turn off Transparent Huge Pages (THP)
Virtual Memory
Reduce swappiness
Turn off reclaim mode (caching effect more important than locality)
![Page 60: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/60.jpg)
SLIDE: 60
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Tuning – Storage Target RAID 10 for best performance
RAID 5 for capacity
Use noop or deadline I/O schedulers
Leverage Tiered Storage
![Page 61: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/61.jpg)
SUMMARY
![Page 62: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/62.jpg)
SLIDE: 62
© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Summary Follow a methodology:
Understand the issue reported
Decide on what data/metrics to collect
Decide how to analyze it
Choose the appropriate tools:
Don’t trust a single tool
Gather metrics incrementally using multiple tools
Tune your targets according to MarkLogic guidelines
![Page 63: Jumpstart: Monitoring and Tuning MarkLogic · 2018. 9. 29. · Primarily GUI -based but can export important data Progressively more powerful and detailed: Procexp (Process Explorer)](https://reader033.vdocuments.net/reader033/viewer/2022051902/5ff2bb1531de3422241a4951/html5/thumbnails/63.jpg)
Q&A