Full monitoring system: Graphite, collectd and StatsD – Part 2

Introduction

In the first part of this series we installed and configured Graphite on an Ubuntu 16.04 server.
Remember that

Graphite

is software for visualizing data, so it’s almost useless if there isn’t a way to collect data and pass it on. This is were 

collectd

become a fundamental part of the full monitoring system: it is a daemon which gathers metrics and makes the information available to the system admin.

In this tutorial we will explore the steps for installing and configuring 

collectd

 to pass data to 

Graphite

.

Collectd Installation

collectd

is available in repositories, so you can install it with 

apt

:


# apt install collectd collectd-utils

That’s it! Now, let’s configure.

Configuring collectd

Edit 

collectd

configuration file:

# $EDITOR /etc/collectd/collectd.conf

In the first lines there is the hostname section. Here, the default file looks like this:

#Hostname "localhost" 
FQDNLookup true

This works fine if there is a real domain name configured. It’s the DNS which gets the proper domain.

Since, in this tutorial,

graphite

and 

collectd

will run on the same machine, uncomment the first line and set a hostname.

On line 42, there is a parameter called Interval which is the time that 

collectd

waits before querying data. Uncomment that line, and change the value from 10 to the shortest interval value chosen in Graphite configuration. In the previous tutorial, it was 20 seconds, so the line will look like this:

Interval 20

The matching of these two values is fundamental. If they do not match some data could be lost!

Now it’s time to choose the services that 

collectd

will gather information about. To do this, collectd uses plugins. Starting on line 88, there is a very long list. Most of the lines are commented, but you’ll want to uncomment apache and write_graphite. After this, the following plugins will be enabled


LoadPlugin apache
LoadPlugin battery
LoadPlugin cpu
LoadPlugin df
LoadPlugin disk
LoadPlugin entropy
LoadPlugin interface
LoadPlugin irq
LoadPlugin load
LoadPlugin memory
LoadPlugin processes
LoadPlugin rrdtool
LoadPlugin swap
LoadPlugin users
LoadPlugin write_graphite

Further down in the file, there are plugins configuration blocks. Uncomment the Apache block and add an Instance sub-block; now the block will look like this:

<Plugin apache> 
#       <Instance "foo"> 
#               URL "http://localhost/server-status?auto" 
#               User "www-user" 
#               Password "secret" 
#               VerifyPeer false 
#               VerifyHost false 
#               CACert "/etc/ssl/ca.crt" 
#               Server "apache" 
#       </Instance> 
# 
#       <Instance "bar"> 
#               URL "http://some.domain.tld/status?auto" 
#               Host "some.domain.tld" 
#               Server "lighttpd" 
#       </Instance> 
 
        <Instance "Graphite"> 
                URL "http://mydomain/server-status?auto" 
                Server "apache" 
        </Instance> 
</Plugin>

Next, configure the 

df

block, which gives information about space occupied on the disk. For example:

<Plugin df> 
        Device "/dev/sda3" 
#       Device "192.168.0.2:/mnt/nfs" 
        MountPoint "/" 
        # FSType "btrfs" 
 
        # ignore rootfs; else, the root file-system would appear twice, causing 
        # one of the updates to fail and spam the log 
        # FSType rootfs 
        # ignore the usual virtual / temporary file-systems 
        # FSType sysfs 
        # FSType proc 
        # FSType devtmpfs 
        # FSType devpts 
        # FSType tmpfs 
        # FSType fusectl 
        # FSType cgroup 
        # IgnoreSelected true 
 
#       ReportByDevice false 
#       ReportInodes false 
 
#       ValuesAbsolute true 
#       ValuesPercentage false 
</Plugin>

Finally, go to the 

write_graphite

section and uncomment it:

<Plugin write_graphite> 
        <Node "graphite"> 
                Host "localhost" 
                Port "2003" 
                Protocol "tcp" 
                LogSendErrors true 
                Prefix "collectd" 
                Postfix "collectd" 
                StoreRates true 
                AlwaysAppendDS false 
                EscapeCharacter "_" 
        </Node> 
</Plugin>

Note: remember that “2003” is the port number on which Carbon will listen.

Save and exit.

Configure Apache

Configure the ‘Apache virtual hosts’ file enabled for Graphite:

# $EDITOR /etc/apache2/sites-available/apache2-graphite.conf

Below the content block, add the following lines:

Alias /content/ /usr/share/graphite-web/static/
    <Location "/content/">
        SetHandler None
    </Location>

    <Location "/server-stats">
        SetHandler server-status
        Require all granted
    </Location>

    ErrorLog ${APACHE_LOG_DIR}/graphite-web_error.log

This means that Apache will serve statistics on the /server-stats page.

Save, exit and reload Apache:

# systemctl reload apache2

Check that everything is well configured by accessing http://localhost/server-stats with a web browser.

Configure storage schemas

To configure the storage schemas correctly, edit the following configuration file:


# $EDITOR/etc/carbon/storage-schemas.conf

In that file, above the default block, insert a new block. For example:

[collectd]
pattern = ^collectd.*
retentions = 20s:1d, 5m:7d, 10m:1y

Save, exit and restart the services:

# systemctl restart carbon-cache
# systemctl restart collectd

Conclusion

Now

collectd

should be up and running, recording all the server activities and flushing these records to

Graphite

.
It’s possible to configure

collectd

with additional plugins, or modify the configuration files to accomplish new/other tasks.

In the last part of this series of tutorials, we will talk about installing and configuring

StatsD

, a program for caching data collected by

collectd

. Stay tuned!