mentat.stats.rrd module

This module contains object encapsulation of RRDTOOL for easier usage in Mentat project.

Overview

In Mentat RRD databases are used to store quick periodical performance statistics. Each database file is configured to store only single dataset and when generating charts multiple RRD files must be combined when necessary to create more complex charts. This approach has the benefit of very easy way of deleting datasets, that are not required anymore, or obsolete and not appearing in life data anymore.

Every RRD database file is configured in following way:

• internal dataset name: eventcnt

• default step: 300 seconds (5 minutes)

• raw storage size: 3 months

• aggregated storage size: 1 year (min, max, avg)

The whole RRD handling functionality is implemented in RrdStats, which receives certain parameters (like locations of database and chart files) upon instantination and those can be then omitted from calls to all other methods. See the documentation below for more details.

There is a definition of color palette in COLOR_PALETTE constant value, which contains all possible colors usable in RRD charts.

The implementation is based on Python library rrdtool (PyPI, documentation).

RRD database identifier naming convention

Names of RRD database files are constructed as follows:

type.snake_case_name.rrd

When being processed, each RRD database file name is split at . character, which yields three tokens:

• type: type, category, or group of the dataset

• snake_case_name: escaped name of the dataset, anything [^-_a-zA-Z0-9] is replaced with _

• file suffix

The idea behind the naming scheme is, that statistics are being calculated separatelly for multiple message groups, for example according to the name of the detector, categories, etc. The type token should uniquelly indentify each of these groups. The snake_case_name then contains underscore-escaped name of the detector/category/etc.

The type.snake_case_name pair is called db_id in further documentation and it is unique identifier of the dataset.

Example usage

>>> rrd_stats = mentat.stats.rrd.RrdStats('/var/tmp', '/var/tmp')
>>> stats.find_dbs()
defaultdict(<class 'list'>, {})

>>> rrd_stats.prepare_db('typea.test_a')
('/var/tmp/utest_rrdstats/typea.testa.rrd', True)
>>> rrd_stats.prepare_db('typea.test_b')
('/var/tmp/utest_rrdstats/typea.testb.rrd', True)
>>> rrd_stats.prepare_db('typeb.test_a')
('/var/tmp/utest_rrdstats/typeb.testa.rrd', True)
>>> rrd_stats.prepare_db('typeb.test_b')
('/var/tmp/utest_rrdstats/typeb.testb.rrd', True)
>>> stats.find_dbs()
defaultdict(<class 'list'>, {'typeb': [('typeb.test_a', 'typeb', 'test_a', '/var/tmp/typeb.test_a.rrd', False), ('typeb.test_b', 'typeb', 'test_b', '/var/tmp/typeb.test_b.rrd', False)], 'typea': [('typea.test_a', 'typea', 'test_a', '/var/tmp/typea.test_a.rrd', False), ('typea.test_b', 'typea', 'test_b', '/var/tmp/typea.test_b.rrd', False)]})

class mentat.stats.rrd.RrdStats(rrds_dir, reports_dir, step=300)

Bases: object

Class implementing RRD database file management and chart generation.

static clean(name)

Cleanup given name so it can be used as RRD database file name.

Parameters:

name (str) – Name to be cleaned.

Returns:

Name with all forbidden characters escaped with _.

Return type:

str

export(ds_id)

Export whole RRD database into JSON structure.

Parameters:

ds_id (str) – Identifier of the dataset.

Returns:

Absolute filesystem path to the prepared RRD database file as string, boolean flag marking creation of new fileand axported dataset as JSON structure.

Return type:

tuple

find_dbs(flt_type=None)

Find all currently available RRD database files.

Parameters:

flt_type (str) – Dataset type filter to dump part of the full result.

Returns:

Structured dictionary containing list of RRD database metadata structures for each type.

Return type:

collections.defaultdict

generate(tst, flt_type=None)

Generate all available RRD charts, spark charts and JSON export files.

Parameters:

• tst (int) – Upper time boundary as unix timestamp.

• flt_type (str) – Dataset type filter to dump part of the full result.

Returns:

List of generated files.

Return type:

list of str

lookup(flt_type=None)

Lookup all available RRD charts, spark charts and JSON export files.

Parameters:

flt_type (str) – Dataset type filter to dump part of the full result.

Returns:

List of available charts.

Return type:

list of dict

prepare_db(ds_id, time_start=None)

Ensure existence of appropriate RRD database file for storing single data set.

Parameters:

• ds_id (str) – Identifier of the dataset.

• time_start (int) – Name to be cleaned.

Returns:

Absolute filesystem path to the prepared RRD database file as string and boolean flag marking creation of new file.

Return type:

tuple

update(ds_id, value, tst=None)

Update given RRD database.

Parameters:

• ds_id (str) – Identifier of the dataset.

• value (int) – Value to be set.

• tst (int) – Update timestamp.

Returns:

Absolute filesystem path to the prepared RRD database file as string and boolean flag marking creation of new file.

Return type:

tuple

update_all(value, tst=None, flt_type=None)

Updade all RRD database files with given value.

Parameters:

• value (int) – Value to be set.

• tst (int) – Update timestamp.

• flt_type (str) – Dataset type filter to dump part of the full result.

Returns:

List of updated RRD database files.

Return type:

list of tuples

exception mentat.stats.rrd.RrdStatsException(description)

Bases: Exception

Base class for all RrdStats specific exceptions.

The rrdtool module does not provide enough granularity with its exceptions and it is necessary to parse the exception string to determine the exact problem. This module provides custom set of exceptions to diferentiate between different problems.

exception mentat.stats.rrd.RrdsCreateException(description)

Bases: RrdStatsException

Exceptions occuring during creating of RRD database files.

exception mentat.stats.rrd.RrdsExportException(description)

Bases: RrdStatsException

Exceptions occuring during exporting of RRD database files.

exception mentat.stats.rrd.RrdsGraphException(description)

Bases: RrdStatsException

Exceptions occuring during generating of RRD charts.

exception mentat.stats.rrd.RrdsUpdateException(description)

Bases: RrdStatsException

Exceptions occuring during updating of RRD database files.