trace compass past present - eclipsecon 2020 › sites › default › files › slides ›...
Post on 24-Jun-2020
5 Views
Preview:
TRANSCRIPT
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 1
Trace Compass PastPresentFuture
Bernd Hufmann, Ericsson AB
2018-10-25
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 2
Agenda
— Background
— Trace Compass History
— Use Cases
— Outlook
— Q & A
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 3
Background
Eclipse Trace Compass is an open source application to solve performance and reliability issues by reading and analyzing traces and logs of a system.
Eclipse Trace Compass is a framework for creating custom applications to solve performance and reliability issues of a system.
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 4
Background
Eclipse Trace Compass is an open source application to solve performance and reliability issues by reading and analyzing traces and logs of a system.
Eclipse Trace Compass is a framework for creating custom applications to solve performance and reliability issues of a system.
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 5
Trace Compass history
2009
Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable test/XML parsers— Trace Research Project
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 6
Trace Compass history
2009
2012
LTTng 2.0— Common Trace Format (CTF)— LTTng 2.0 Kernel and UST— LTTng Tracer Control
Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable test/XML parsers— Trace Research Project
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 7
Trace Compass history
2009
2012
Standalone Application (RCP)— Analysis Framework— New views types— More trace types (e.g. GDB Trace)
LTTng 2.0— Common Trace Format (CTF)— LTTng 2.0 Kernel and UST— LTTng Tracer Control
20131
Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable test/XML parsers— Trace Research Project
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 8
Trace Compass history
2009
2012
Standalone Application (RCP)— Analysis Framework— New views types— More trace types (e.g. GDB Trace)
LTTng 2.0— Common Trace Format (CTF)— LTTng 2.0 Kernel and UST— LTTng Tracer Control
20131
Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable text/XML parsers— Trace Research Project
2014— Own Eclipse Project— More trace types (not only Linux)— Time synchronization of traces
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 9
Trace Compass history
2009
2012
Standalone Application (RCP)— Analysis Framework— New views types— More trace types (e.g. GDB Trace)
LTTng 2.0— Common Trace Format (CTF)— LTTng 2.0 Kernel and UST— LTTng Tracer Control
20131
Joining Eclipse Linux Tools— Tracing and Monitoring Framework— Support for LTTng 0.x— Customizable test/XML parsers— Trace Research Project
2014— Own Eclipse Project— More trace types (not only Linux)— Time synchronization of traces 2015 Performance Analysis
— Latency and real-time analysis— Pattern Analysis— Critical Path
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 10
Trace Compass history
2016
Call Graph Analysis— Flame Graph View— Function Duration Statistics— Function Duration Density
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 11
Trace Compass history
2016
Trace Compass Add-ons— More trace types
— ftrace, atrace, maven, trace event— Many different views, e.g.
— Generic Flame Chart/ Graph— VM analysis views
Call Graph Analysis— Flame Graph View— Function Duration Statistics— Function Duration Density
2017
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 12
Trace Compass history
2016
Trace Compass Add-ons— More trace types
— ftrace, atrace, maven, trace event— Many different views, e.g.
— Generic Flame Chart/ Graph— VM analysis views
Call Graph Analysis— Flame Graph View— Function Duration Statistics— Function Duration Density
2017
2018Jaeger Tracing— Trace parser for Jaeger traces— Various views for Jaeger traces— Container Analysis— In Trace Compass incubator project
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 13
Trace Compass history
2016
Trace Compass Add-ons— More trace types
— ftrace, atrace, maven, trace event— Many different views, e.g.
— Generic Flame Chart/ Graph— VM analysis views
Call Graph Analysis— Flame Graph View— Function Duration Statistics— Function Duration Density
2017
2018Jaeger Tracing— Trace parser for Jaeger traces— Various views for Jaeger traces— Container Analysis— In Trace Compass incubator project Distributed Architecture
— Trace Server Protocol (TSP)— Trace Compass trace server— Web-frontend
2019
2020
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 14
Collaboration with academia
— Trace Research Project ongoing over many years
— University “Polytechnique Montréal” and others
— Researchers and students contribute their outcome to the
— mainline Trace Compass project
— Trace Compass Incubator Project
— ~30% of commits come from “Polytechnique Montréal”
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 15
Data Flow
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 16
Data Flow
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 17
— Searching
— Filtering
Events Table
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 18
— Searching
— Filtering
Highlighting
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 19
— Searching
— Filtering
— Highlighting
Searching and Filtering
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 20
— Searching
— Filtering
Bookmarking
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 21
System tracing
— Linux Tracing Toolkit next generation (LTTng)
— Kernel analysis
— Static tracepoints of the Linux Kernel
— Same tracepoints used by Perf or Ftrace which populate same views
— User Space Tracing (UST)
— Static tracepoints inserted in user application
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 22
System tracing Linux kernel analysis
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 23
Visual grepPress key: /
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 24
Filtering
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 25
Resource usage
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 26
System tracing highlights
— Follow CPU and IRQ states
— Follow thread execution
— Find high processor load
— Find memory usage issues
— Many views available analyzing Linux kernel traces
— Enhanced highlighting and filtering
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 27
— Define a state machine for timing analysis
— Implementation in Java as Trace Compass extension
— Data-driven pattern matching (in XML)
— Defining timing analyses on-the-fly
— Measure between start and end
— Store in a built-in segment store
— Visualize data in various views
Latency analysis
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 28
Latency analysis views
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 29
Critical path
— Displays of system wait chains for given process across network
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 30
Latency analysis highlights
— Find long latencies
— Real-time deadline investigation and verification
— Follow critical path execution
— Performance investigations
— Customizable analysis
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 31
Call graph analysisFlame Chart (time based)
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 32
Call graph analysisFlame Graph (aggregation)
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 33
Call graph analysisFunction duration statistics
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 34
Call graph analysis highlights
— Profiling using trace data
— Flame Graph and other views
— Find long executions
— Performance investigation
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 35
Trace Compass add-ons
— More analyses
— Call stack analyses
— Trace Types (e.g. Perf, Ftrace, jaeger)
— Virtualization (e.g. VM analysis)
— How to install
— Menu “Tools -> Add-ons…”
— Update Site
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 36
OpenTracing API Jaeger tracing support
— Traces according the OpenTracing API
— Implemented to trace Microservices
— Follow flows and spans across services
— Jaeger traces content:
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 37
Jaeger trace import
— JSON file directly
— Via HTTP request
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 38
Span live viewLogs
Spans
Logs Tooltip
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 39
Trace Compass Jaeger highlights
— Correlation with system and application traces
— Drill-down into spans
— Support of large traces
— Easy navigation and zooming
— Other views available
— Possibility to use critical path analysis to reduce spans
— Possibility of custom analysis (XML)
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 40
Trace Compass architecture
Trace Compass Core
Trace Compass UI
Trace State
System
Segment
store
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 41
Trace Compass architecture
Trace Compass Core
Trace Compass UI
Trace State
System
Segment
store
Trace Compass Core
Trace Compass UI
Trace State
System
Segment
store
Data Providers
Current (ongoing)
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 42
Server-client architectureProposal
Trace Compass Core
Trace Compass UI
Trace State
System
Segment
store
Trace Compass Core
Trace Visualization Client
Trace State
System
Segment
store
Trace Server Protocol (REST or RPC)
Data Serializer
Data Providers
Presentation Layer
Business Layer
Data Layer
TraceServer
Data Providers
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 43
— Handle the communication between core and UI of trace viewer
— Exchange visualization data between a client and a server
— Trace management
— Various visualization types
— Server side filtering and searching
— Not tracer control
— Proof-of-concept prototype done
— https://github.com/theia-ide/trace-server-protocol
— Currently build for REST over HTTP
— Trace server in Trace Compass incubator project
Scope of Trace Server Protocol (TSP)
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 44
Distributed architecture highlights
— Leveraging modern UI technologies and other programming languages
— Better integration with existing web-based services
— CI integration (e.g. open traces directly from Jenkins)
— TR reporting tool (e.g. open traces)
— Integration with workspace management applications (e.g. Che)
— Integration with web-based IDEs
— Leveraging language and debug server (LSP/DAP)
— Allows for usage of cloud/container based technologies (e.g. docker)
— Enables Micro-services architecture
— Distributed services and trace analyses
— Higher scalability
— Easier to change services due decoupled micro-services architecture
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 45
— Project pages
— http://tracecompass.org
— http://projects.eclipse.org/projects/tools.tracecompass
— http://projects.eclipse.org/projects/tools.tracecompass.incubator
— Documentation
— Trace Compass User Guide
— Trace Compass Developer Guide
— Trace Research Projects
— http://hsdm.dorsal.polymtl.ca/
— http://amdls.dorsal.polymtl.ca/
References
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 46
— Presenter
— Bernd Hufmann: Bernd.Hufmann@ericsson.com
— Mailing list
— tracecompass-dev@eclipse.org
— IRC
— oftc.net #tracecompass
— Mattermost
— https://mattermost-test.eclipse.org/eclipse/channels/trace-compass
Contacts
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 47
Ericsson Internal | 2018-02-212018-10-25 | Ericsson 48
Q & A
Characters for Embedded characters:!"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČčĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fiflΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№—–-
!"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČčĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fiflΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№—–-
!"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČčĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fiflΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№—–-
!"#$%&'()*+,./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČčĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fiflΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№—–-
top related