Understanding Blackboard Learn

Error Logs

Matthew Saltzman

Security EngineerBlackboard Inc.

Goals

• During his presentation, we should cover the following topics

• Configuring the different Blackboard LearnTM log files

• Where different errors will log to

• How to read the Blackboard Learn log files

• Determining which errors are cause for concern

• Where to start looking for errors relating to logged messages

2

Why Bother?

• Reading logged errors in an enterprise application is difficult and tedious

• Errors could be expected, and not cause concern

• Could be related to bugs that can't be fixed without help from support

• Most importantly, it creates more work to do

3

Reasons

• Help spot errors before users, and thus close help desk tickets faster

• Have your support tickets resolved faster, as support will spend less time looking for the right log file

• Help prevent performance issues

• Keep the Blackboard Learn system running smoothly

4

5

Error Messages

Explained

Where are they?

• All log files, save Building BlockTM logs, are in blackboard/logs

• Or a sub directory

• Each log file has information from a specific task

• Also, each log is set to a different log level

• FATAL, ERROR, WARNING, INFORMATIONAL, DEBUG are the most common

6

7

Any other way?

• Other ways to find log files

• Download from the admin panel

• Admin console

• Sometimes in the frame source when an error occurs

8

9

Log Configuration

• These log files have different configuration files, though in some cases multiple logs are configured in a single config file

• For most, config files are located in blackboard/config/*.properties

• Stdout-stderr-log.txt is configured in the blackboard/config/tomcat/conf/wrapper.conf file

• Configuration is somewhat labeled inside the file

10

service-config.properties

• Common log configuration file

• Holds configuration information for bb-services-log.txt and bb-sqlerr-log.txt

• On the right, showing where bb-services-log.txt is configured

11

wrapper.conf

• Familiar config file for tomcat service wrapper

• Also contains log configuration for stdout-stderr-log.txt

• Right hand side shows the logs' configuration

12

What goes Where?

• Sometimes hard to tell

• Most commonly, the errors are logged to bb-services, bb-sqlerr, or stdout-stderr

• Bb-services log holds webapp errors

• For example, errors found in the UI

• Bb-sqlerr log holds only database errors, such as Ora-codes

• Stdout-stderr contains errors that reach the standard error stream, as well as startup and shutdown information

13

Example Stack Trace

com.inet.tds.Tds4SQLException: Msg 6005, Level 14, State 2, Line 1,

Sqlstate 01000[*****************]SHUTDOWN is in progress.

at com.inet.tds.aa.a(Unknown Source)

at com.inet.tds.n.b(Unknown Source)

at com.inet.tds.n.a(Unknown Source)

at com.inet.tds.n.a(Unknown Source)

at com.inet.tds.bg.g(Unknown Source)

at com.inet.tds.bg.executeQuery(Unknown Source)

at com.inet.pool.u.executeQuery(Unknown Source)

at com.inet.pool.ac.executeQuery(Unknown Source))

14

Parts of Stack Trace

• Stack traces can be broken down into smaller components

• Error Message, contains any text about the error, as well as the exception that was caught

• Thread stack showing the path the thread took to reach the error

• Optional caused by or root cause clauses, showing deeper errors and associated thread stacks

15

Exception/Message

• First line of the stack trace

• Begins with a java exception

• Additional error details

• Sometimes contains an error id, which can link to what a user sees in the UI

• Example:

com.inet.tds.Tds4SQLException: Msg 6005, Level 14, State 2, Line 1, Sqlstate 01000[*****************]SHUTDOWN is in progress.

16

Thread Stack

• Each line begins with the word "at"

• Provides the full class name that threw the exception, complete with package details

• Specific line number in the code where the exception occurred

17

1818

Optional Sections

• Begins with either the words root cause or caused by

• Contains another exception and stack trace

• Shows where an earlier, causal exception occurred in the code

• Example:

19

Other types of Logs

• Access_logs from apache, IIS, tomcat, containing request information

• Email logs simply showing what emails went out, and to whom

• Authentication and SIS integration logs, visible in the GUI

20

21

22

Troubleshooting

Error Messages

Important Errors

• Blackboard Learn logs tons of exceptions and informational messages, so how does one know which to pay attention to?

• Errors users see in the UI

• Errors pointing out performance issues

• Errors describing non-working functionality

• "Log Spam", eg errors that do not affect behavior of the system, and informational messages

23

24

Errors seen by Users

• Any time a user sees an error, it's something to investigate

• In most cases, users should not see exceptions

• Always display error ids, unique alphanumeric values that can pinpoint an error in a log

25

26

Performance Errors

• These are errors indicating that the entire application is not processing requests normally

• HeapSpaceOutOfMemory is only one example

• Deadlock errors

• Hot Spot JVM errors

• Consecutive full garbage collections

27

28

Broken Functionality

• Much tougher to identify

• Usually calls out a specific blackboard component

• Could come in many forms

• Null Pointer Exception

• Class Not Found exception

• Authorization errors

29

Broken Functionality

• Need to be investigated before conclusions are drawn

• Can use the access logs to find the user that reached the error page at the right time, and work through their path through the system to reach that page

• Also can use the stack trace and error message to guess what the user was doing

• Usually involve data corruption of some type, or a bug in Blackboard Learn

30

Log Spam

• Don't appear to correlate to a specific action in Blackboard Learn

• Errors are generally generic and obscure

• Messages are also repetitive

• But since users are unaffected, relatively few trouble tickets come in

31

Summary

• We should have covered the basics of troubleshooting blackboard errors

• Should not need a description from a user to find a problem

• Correlate errors and actions within the system

32

33

Questions?