I used to work with Graphite for a while, I can say that it is one of my favorite tools in devops cultural movement where spreading operation's awareness among operation, development teams and other stakeholders is key factor.
Graphite stores time series data and provides powerful set of functions to manipulate with data, which is very useful for learning about trend of resource consumption as well finding problematic one among large number physical/logical resources.
Installation
In general installation is not difficult. To set it up just follow the installation document.
How it work
Graphite is written entirely in Python. From run time point of view Graphite consist of 1) Carbon Cache (developed using Twisted) and 2) WebApp (developed using Django). Metric data are stored in file system , one metric per file in a format designed specially for time series data.
We get the graph by sending a HTTP request to URL of the WebApp. We use parameters of URL to tell Graphite what kind of graph of which time period we want. We can not only request graph of ordinary metrics but by using Graphite functions also to combine, filter, transform these metrics to generate rich, powerful graphs useful for planning and troubleshooting.
The WebApp read data of metrics involved from corresponding per metric data file and also from Carbon Cache for those values that have not yet written to the data file.
We send metrics to the Carbon Cache process, which is responsible for writing data into data file and serving query from WebApp. To scale up, many Carbon Caches may be configured behind a Carbon relay.
Getting Metrics into Graphite
It is pretty simple, just send metric in form of metric_name metric_value seconds_from_epoch to default Graphite Carbon default port 2003 e.g.
There is no need to create metric definition in advance, just send data to Graphite and it will create a metric for you if it does not yet exists. Graphite uses pattern of metric name defined in storage-schemas.conf to determine how often (in term of seconds) one data point of certain metric is stored. For example if the rule say that a metric is once per minute and we send two values of the same metric at the same minute then the second one will overwrite the first one. It is also obvious that Graphite does not support finer resolution than one second.
Metric name can be composed of node separated by dot which allows us to organize metrics into hierarchy enabling combination them using Graphite functions. Suppose we have N httpd servers and we want to track their consumption of network bandwidth in term of megabyte incoming and outgoing each minute, we can use the following naming convention
Asking questions
There may questions that can be asked about the system. Given the example above, we can draw a graph of total of in coming or network bandwidth per min in mega byte by enter function sumSeries
To give a nice legend to a graph we can decorate original expression with function alias
Every time series data is a trending line however , I feel that some time it is useful to draw two lines one current and other from the past. So we see the difference over period of time let say one day. The function for doing that is timeShift. Let compare the difference between today traffic with same day last week.Because volume of traffic depends on day of week (weekend traffic is much lower), it is good idea compare one graph with other on the same week day.
Compare different metrics may be useful in case we want to know why e.g. value of a high level metric (e.g. response time of an URL) now is higher than in the past. Try to relate the high level metric to several low level metrics can help use to pinpoint where is the issue. Because different metrics has different scale the function secondYAXIS
Performance hacks
Having dozens of dashboards with hundreds of graphs each requires hundreds or thousands of metrics when rendering can bring down the Graphite server to knee. The following hacks may be helpful for such situation.
When rendering a graph, WebApp identify metrics involved and for each metric WebApp find the corresponding file name, read its content of the file then query the metric data from Carbon Cache finally merges the two result.
As cost of remote query to Carbon Cache is non negligible, a typical requested graph requiring hundreds of metrics will result in the same number of remote query being sent to Carbon Cache, which consumes substantial amount CPU. To mitigate it I have patched both WebApp and Carbon Cache of Graphite version 0.9.x so WebApp will send a bulk of request for metrics in single call to Carbon Cache see my commits on github
- https://github.com/huy/graphite-web/commit/86d1d2628643843aae9649d5187454cef0c1322b,
- https://github.com/huy/carbon/commit/2758c68d6698d20316af8da27f11d6383f0f59a0
Both Carbon Cache and WebApp generate a lot of IO. If the IO system is not fast enough, the server may sometimes hang waiting for completion of IO. We can improve the situation a bit by tuning the file system where Graphite stores their metrics files. My favorite options when mounting the file system is
The noatime option instruct file system not to change access time (changing access time modify disk block containing inode) of a file when someone read it. Other options trade off safety of file system for performance see ext3 document for detail.
References
References
