.. _section-reporting: Reporting ================================================================================ Reporting is a vital part of Mentat system. Currently there are following two types of reporting implemented: Event reporting Periodical reports generated for each abuse group and severity. Statistical reporting Periodical informational reports about overall performance of Mentat system. Event reporting -------------------------------------------------------------------------------- Event reporting feature is implemented within the :ref:`mentat-reporter.py ` module. This module is intended to enable distribution of events in the form of periodical reports to appropriate users. These users are mostly administrators within networks for which the Mentat system has relevant data. Report types ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Currently Mentat system supports following types of event reports: **summary** This type of report contains all relevant data for particular network formated neatly into one email. **extra** This type of report contains all relevant data for particular network formated neatly into one email, but regarding only single IP address. This type can be prefered in case respondents are using some kind of ticket system and want to resolve the issues separately for each host. This means, that when there are 10 events regarding 10 different hosts, this will produce 10 separate email reports. Report emails ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ All report emails sent by Mentat :ref:`mentat-reporter.py ` have following common features: * Report message in human readable format. * Various custom email headers to enable automated processing and filtering. * Metadata summary in report footer. The human readable message organizes reported events into sections containing events grouped according some internal classification. For example each detector may describe SSH bruteforce attack with different IDEA message. Mentat attempts to classify these messages into the same class and display them in the same section. The report message however attempts to keep things as simple as possible and be as brief as possible, so only source address, event time window and event count are displayed in the table in each section. For more information about the possible problem you have to look into the attached file containing raw data (or download it from the server). Report emails contain following usefull headers, that can be used for automated report processing and filtering: .. warning:: Please note the deprecation of some of the email headers from previous version of Mentat :ref:`mentat-reporter.py ` and their new names. **X-Mentat-Report-Class** Class of the current report. For periodical event reports this header has fixed value ``events``. **X-Mentat-Report-Id** Unique report identifier. In previous version of Mentat :ref:`mentat-reporter.py ` this header was called **X-Cesnet-Report-Id**. This should now be considered **deprecated** and you should not use it anymore, support for old header will be removed in one of the future versions. **X-Mentat-Report-Parent-Id** Unique identifier of parent summary report for ``extra`` reports. **X-Mentat-Report-Type** Type of the current event report. It can contain one of the following values: ``summary``, ``extra``. In previous version of Mentat :ref:`mentat-reporter.py ` this header was called **X-Cesnet-Report-Type**. This should now be considered **deprecated** and you should not use it anymore, support for old header will be removed in one of the future versions. **X-Mentat-Report-Severity** Severity of the current report. It can contain one of the following values: ``low``, ``medium``, ``high``, ``critical``. In previous version of Mentat :ref:`mentat-reporter.py ` this header was called **X-Cesnet-Report-Severity**. This should now be considered **deprecated** and you should not use it anymore, support for old header will be removed in one of the future versions. **X-Mentat-Report-Src-IP** This header contains IP adress appropriate for ``extra`` reports. In previous version of Mentat :ref:`mentat-reporter.py ` this header was called **X-Cesnet-Report-Srcip**. This should now be considered **deprecated** and you should not use it anymore, support for old header will be removed in one of the future versions. **X-Mentat-Report-Event-Count** Total number of events reported in current..ehm...report. **X-Mentat-Report-Time-Window** Time window for current report as two timestamps in `ISO 8601 `__ format (YYYY-MM-DDTHH:MM:SS) concatenated by ``___`` string. Example: ``2018-07-13T10:30:00___2018-07-13T10:50:00`` **X-Mentat-Report-Test-Data** Flag indicating whether this report contains test or experimental data. It can contain one of the following values: ``True``, ``False``. Report always contains either only experimental, or only production data, there can never be a mix of any kind. When this value is set to ``True``, you may safely ignore current report in production environment. Report detail in web interface ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ All generated reports are stored on the server and accessible for the registered users. Accessing the reports via web interface provides users with additional options and features. Reporting algorithm ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. note:: Work in progress. Reporting customization for users ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. note:: Work in progress. Reporting configuration for administrators ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default the reporter is configured to prevent from unwanted spamming of target abuse contacts. This is great for initial deployment, but you have to make extra effort to really start sending emails. Please continue reading for more details. Most important configuration files and directories related to reporting: /etc/mentat/core/common.json.conf Common application configurations. Pay attention to ``mail_`` settings, because they can override the reporting settings for abuse groups set in database. This approach is designed to prevent a recently installed Mentat system from spamming real world abuse contacts. So by default all reports are redirected to local ``root`` contact. To enable reporting in production environments it is necessary to disable the ``mail_to`` setting in this file and to enforce ``mail_to: null`` in the main ``/etc/mentat/mentat-reporter.py.conf`` configuration file. /etc/mentat/core/reporting.json.conf Common reporting configurations. Contains paths to templates, ``mail_admin`` directive and ``template_vars`` object with data supplied to reporting templates. /etc/mentat/mentat-reporter.py.conf Main configuration file of reporting component. Tweak this configuration file to customize the bahaviour of the reporting module itself. The most interesting options are customizations of various report email headers and report template customizations. Important: to enable reporting in production environment it is necesssary to set ``mail_to: null`` in this file and to disable its default value (``root``) in ``/etc/mentat/core/common.json.conf`` configuration file (see above). Otherwise each report will be forcefully redirected to local ``root`` to prevent Mentat system from spamming abuse contacts. /etc/mentat/templates/reporter/default.summary.txt.j2 Jinja2 template defining content of *summary* event reports. This template is used when rendering summary event reports. It is completely localized, so it can be easily translated to additional languages. It is prepared for minor tweaks. For bigger content adjustments you have to create your own template and select it with ``template_id`` directive in ``/etc/mentat/mentat-reporter.py.conf`` configuration file. /etc/mentat/templates/reporter/default.extra.txt.j2 Jinja2 template defining content of *extra* event reports. This template is used when rendering summary event reports. It is completely localized, so it can be easily translated to additional languages. It is prepared for minor tweaks. For bigger content adjustments you have to create your own template and select it with ``template_id`` directive in ``/etc/mentat/mentat-reporter.py.conf`` configuration file. Event classes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Reporting is based on event classes. Event classes are an internal classification of events in Mentat. Usually, IDEA categories are split into one ore more event classes for better reporting. They are assigned by mentat-inspector according to event class settings in the web GUI. If you want to manage event classes, please look at the documentation of the :ref:`event class module `.