Download - CAVE Overview
CAVE - an overviewVal Dumitrescu
Paweł Raszewski
At GILT:● OpenTSDB + Nagios● DataDog● NewRelic● Notifications with PagerDuty
Another monitoring solution?
ContinuousAudit
VaultEnterprise
What is CAVE?
A monitoring system that is:● secure● independent● proprietary● open source
What is CAVE?
● horizontally scalable to millions of metrics, alerts● multi-tenant, multi-user● extensible HTTP-based API● flexible metric definition● data aggregation / multiple dimensions● flexible and extensible alert grammar● pluggable notification delivery system● clean user interface for graphing and dashboarding
Requirements
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Alert Grammar
Metric has name and tags (key-value pairs)e.g.orders [shipTo: US]response-time [svc: svc-important, env: prod]
Alert Grammar
Aggregated Metric has metric, aggregator and period of aggregation, e.g.orders [shipTo: US].sum.5mresponse-time [svc: svc-important, env: prod].p99.5m
Supported aggregators:count, min, max, mean, mode, median, sumstddev, p99, p999, p95, p90
Alert Grammar
Alert Condition contains one expression with two terms and an operator. Each term is a metric, an aggregated metric or a value.e.g.orders [shipTo: US].sum.5m < 10orders [shipTo: US].sum.5m < ordersPredictedLow [shipTo: US]
Alert Grammar
An optional number of times the threshold is broken, e.g.response-time [svc: svc-team, env: prod].p99.5m > 3000 at least 3 times
Alert Grammar
Special format for missing datae.g.orders [shipTo: US] missing for 5mheartbeat [svc: svc-important, env: prod] missing for 10m
Alert Grammartrait AlertParser extends JavaTokenParsers { sealed trait Source case class ValueSource(value: Double) extends Source case class MetricSource( metric: String, tags: Map[String, String]) extends Source case class AggregatedSource( metricSource: MetricSource, aggregator: Aggregator, duration: FiniteDuration) extends Source
sealed trait AlertEntity case class SimpleAlert( sourceLeft: Source, operator: Operator, sourceRight: Source, times: Int) extends AlertEntity case class MissingDataAlert( metricSource: MetricSource, duration: FiniteDuration) extends AlertEntity …}
Alert Grammartrait AlertParser extends JavaTokenParsers { … def valueSource: Parser[ValueSource] = decimalNumber ^^ { case num => ValueSource(num.toDouble) }
def word: Parser[String] = """[a-zA-Z][a-zA-Z0-9.-]*""".r def metricTag: Parser[(String, String)] = (word <~ ":") ~ word ^^ { case key ~ value => key -> value }
def metricTags: Parser[Map[String, String]] = repsep(metricTag, ",") ^^ { case list => list.toMap } …}
Alert Grammartrait AlertParser extends JavaTokenParsers { … def metricSourceWithTags: Parser[MetricSource] = (word <~ "[") ~ (metricTags <~ "]") ^^ { case metric ~ tagMap => MetricSource(metric, tagMap) }
def metricSourceWithoutTags: Parser[MetricSource] = word ^^ { case metric => MetricSource(metric, Map.empty[String, String]) }
def metricSource = metricSourceWithTags | metricSourceWithoutTags
…}
Alert Grammartrait AlertParser extends JavaTokenParsers { … def duration: Parser[FiniteDuration] = wholeNumber ~ ("s"|"m"|"h"|"d") ^^ { case time ~ "s" => time.toInt.seconds case time ~ "m" => time.toInt.minutes case time ~ "h" => time.toInt.hours case time ~ "d" => time.toInt.days }
def aggregatedSource: Parser[AggregatedSource] = (metricSource <~ ".") ~ (aggregator <~ ".") ~ duration ^^ { case met ~ agg ~ dur => AggregatedSource(met, agg, dur) }
def anySource: Parser[Source] = valueSource | aggregatedSource | metricSource …}
Alert Grammartrait AlertParser extends JavaTokenParsers { … def missingDataAlert: Parser[MissingDataAlert] = metricSource ~ "missing for" ~ duration ^^ { case source ~ _ ~ d => MissingDataAlert(source, d) }
def simpleAlert: Parser[SimpleAlert] = anySource ~ operator ~ anySource ^^ { case left ~ op ~ right => SimpleAlert(left, op, right, 1) }
def repeater: Parser[Int] = "at least" ~ wholeNumber ~ "times" ^^ { case _ ~ num ~ _ => num.toInt } def simpleAlertWithRepeater: Parser[SimpleAlert] = anySource ~ operator ~ anySource ~ repeater ^^ { case left ~ op ~ right ~ num => SimpleAlert(left, op, right, num) }
Alert Grammartrait AlertParser extends JavaTokenParsers { … def anyAlert: Parser[AlertEntity] = missingDataAlert | simpleAlertWithRepeater | simpleAlert}
Usage:class Something(conditionString: String) extends AlertParser {
… parseAll(anyAlert, conditionString) match {
case Success(SimpleAlert(left, op, right, times), _) => … case Success(MissingDataAlert(metric, duration), _) => … case Failure(message, _) => … }}
Functional Relational Mapping (FRM) library for Scala
Slick <> Hibernate
Slick
compile-time safetyno need to write SQL
full control over what is going on
Slick
Scala Collections APIcase class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
Scala Collections APIcase class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
list.filter(_.id > 1)
Scala Collections APIcase class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
list.filter(_.id > 1).map(_.name)
Scala Collections APIcase class Person(id: Int, name: String)
val list = List(Person(1, "Pawel"),
Person(2, "Val"),
Person(3, "Unknown Name"))
list.filter(_.id > 1).map(_.name)
SELECT name FROM list WHERE id > 1
Schema
ORGANIZATIONS TEAMS
Entity mapping/** Table description of table orgs.*/
class OrganizationsTable(tag: Tag) extends Table[OrganizationsRow](tag,"organizations") {
...
/** Database column id AutoInc, PrimaryKey */
val id: Column[Long] = column[Long]("id", O.AutoInc, O.PrimaryKey)
/** Database column name */
val name: Column[String] = column[String]("name")
/** Database column created_at */
val createdAt: Column[java.sql.Timestamp] = column[java.sql.Timestamp]("created_at")
… /** Foreign key referencing Organizations (database name token_organization_fk) */
lazy val organizationsFk = foreignKey("token_organization_fk", organizationId,
Organizations)(r => r.id, onUpdate = ForeignKeyAction.NoAction, onDelete =
ForeignKeyAction.NoAction)
}
CRUDval organizationsTable = TableQuery[OrganizationsTable]
// SELECT * FROM ORGANIZATIONS
organizationsTable.list
// SELECT * FROM ORGANIZATIONS WHERE ID > 10 OFFSET 3 LIMIT 5
organizationsTable.filter(_.id > 10).drop(3).take(5).list
// INSERT organizationsTable += OrganizationsRow(1, "name", "email", "notificationUrl", ... , None,
None)
// UPDATE ORGANIZATIONS SET name = “new org name” WHERE ID=10
organizationsTable.filter(_.id === 10).map(_.name).update("new org name")
// DELETE FROM ORGANIZATIONS WHERE ID=10
organizationsTable.filter(_.id === 10).delete
Queries - JOINSval organizationsTable = TableQuery[OrganizationsTable]
val teamsTable = TableQuery[TeamsTable]
val name = “teamName”
val result = for {
t <- teamsTable.sortBy(_.createdAt).filter(t => t.deletedAt.isEmpty)
o <- t.organization.filter(o => o.deletedAt.isEmpty && o.name === name)
} yield (t.name, o.name)
SELECT t.name, o.name FROM TEAMS t
LEFT JOIN ORGANIZATIONS o ON t.organization_id = o.id
WHERE t.deleted_at IS NULL AND o.deleted_at IS NULL AND o.name = `teamName`
ORDER BY t.created_at
SELECT t.name, o.name FROM TEAMS t
LEFT JOIN ORGANIZATIONS o ON t.organization_id = o.id
WHERE t.deleted_at IS NULL AND o.deleted_at IS NULL AND o.name = `teamName`
ORDER BY t.created_at
val result: List[(String, String)]
Connection pool and transactionsval ds = new BoneCPDataSource
val db = {
ds.setDriverClass(rdsDriver)
ds.setJdbcUrl(rdsJdbcConnectionString)
ds.setPassword(rdsPassword)
ds.setUser(rdsUser)
Database.forDataSource(ds)
}
db.withTransaction { implicit session =>
// SLICK CODE GOES HERE
}