Reporting

Blip does not have a single format or protocol for reporting metrics. Instead, after collecting metrics Blip uses metric sinks to translate the Blip metric data structure to a sink-specific protocol.

Normally, a sink sends metrics somewhere, but a sink can do anything with the metrics—Blip does not impose any restrictions on sinks. For example, the default log sink dumps metrics to STDOUT.

Blip has several built-in sinks, but unless you happen to use one of them, you will need to write a custom sink to make Blip report metrics in your environment. Start by understanding how Blip reports metrics internally (as detailed on this page), then read Develop / Sinks.

Blip metric types are standard:

• COUNTER
• GAUGE
• BOOL
• EVENT (reserved for future use)

Blip automatically uses the correct metric type for all metrics.

Blip, like MySQL, does not distinguish between “counter” and “cumulative counter”. Blip counter metrics can reset to zero if MySQL is restarted; otherwise, the value only increases.

All values, regardless of type, are float64.

Negative values are allowed. Some derived metrics use negative values as documented in the domain reference. MySQL metrics are not supposed to be negative, but there are MySQL bugs that cause negative values.

MySQL metrics use a variety of units—from picoseconds to seconds. When the MySQL metric unit is documented and consistent, Blip reports the value as-is. For example, innodb.buffer_flush_avg_time is documented as “Avg time (ms) spent for flushing recently.”, therefore Blip reports the value as-is: as milliseconds.

When the MySQL metric unit is variable, Blip uses the following units:

For example, query response time ranges from nanoseconds to seconds (with microsecond precision) depending on the source. But regardless of the source, Blip reports query time as microseconds (μs).

To convert units, use the TransformMetrics plugin or write a custom sink

Blip does not suffix metric names with units, and it does not strip the few MySQL metrics that have unit suffixes.

Blip never renames MySQL metrics on collection or within its metric data structure. Metrics can be renamed after collection by using the TransformMetrics plugin or writing a custom sink.

Internally, Blip stores metrics in a Metrics data structure:

type Metrics struct {
	Begin     time.Time                // when collection started
	End       time.Time                // when collection completed
	MonitorId string                   // ID of monitor (MySQL)
	Plan      string                   // plan name
	Level     string                   // level name
	State     string                   // state of monitor
	Values    map[string][]MetricValue // keyed on domain
}

Metric values (the last field in the struct) are reported per-domain. To visual the metric values data structure, in YAML it would be:

status.global: # domain

  - name:  queries
    value: 5382      
    type:  1 # counter
    group:
      key1: "val1"
    meta:
      key1: "val1"

  - name: threads_running
    value: 15
    type:  2 # gauge

Blip metrics are not stored or reported in YAML. Examples are for illustration only.

All metrics belong to a single domain, and each domain can collect any number of metrics.

All metrics have a name, type, and value. Blip automatically sets the type and value of every metric. The name is set by MySQL (for MySQL metrics) or Blip (for derived metrics). Blip lowercases all metric names, even MySQL metric names.

Some metrics have groups and meta.

Metric sinks convert the metric data structures to a sink-specific data structure. For example, Prometheus emulation acts as a pseudo-sink to convert and report Blip metrics in Exposition format:

# TYPE mysql_global_status_threads_running gauge
mysql_global_status_threads_running 16
# TYPE mysql_global_status_queries counter
mysql_global_status_queries 138923

After Blip sends metrics to a sink, the sink fully owns the metrics; Blip no longer references the metrics.

Certain domains set group key-value pairs to distinguish different metrics with the same name. For example, the size.database domain groups metrics by database name. Each metric is grouped on db:

size.database:
  - bytes:
      value: 50920482048
      type: 2 # gauge
      group:
        db: "foo"
  - bytes:
      value: 8920482048
      type: 2 # gauge
      group:
        db: "bar"
  - bytes:
      value: 59840964096
      type: 2 # gauge
      group:
        db: "" # all databases

The metric name is the same—bytes—but there are 3 instances of the metric for each of the 3 groups: db=foo, db=bar, and db="". The last group, db="", is how this domain represents the metric for all databases (the global group).

When set, groups are required to uniquely identify metrics. In the example above, the derived metric name bytes does not unique identify a metric; you must include the group keys. How the group keys are included is sink-specific, but most translate the groups to tags, labels, or dimensions.

Groups, labels, and dimensions serve the same purpose. Blip uses the term group because it’s similar to MySQL GROUP BY.

Certain domains set metadata about metrics as documented. An example is query.response-time:

query.response-time:
  - p95:
      value: 500.2
      type: 2 # gauge
      meta:
        p95: "95.8"

The domain collects the P95 query response time value, but due to how MySQL calculates percentiles, it might not be the exact P95. In the metadata, the collector adds the collected percentile (p95) and the real percentile value: 95.8.

Another example is repl: it sets meta key source equal to Source_Host (or Master_Host) from SHOW REPLICA STATUS.

Unlike groups, meta does not uniquely identify metrics and is optional.