“mtail” (https://github.com/google/mtail) is so called “whitebox log monitoring solution. But its usage is not exactly straight forward so here are some small hints:

Installation (build):

  • install GO following
  • set GOPATH variable (like $HOME/go)
  • create subdirectories – mdir -p $GOPATH/src/github.com/google
  • cd $GOPATH/src/github.com/google
  • clone mtail in this directory
  • cd mtail
  • make
  • go install
  • test installation “mtail –help” – if you get help message it is great

Programs for mtail:

  • mtail automatically ads 2 tags to the metrics – hostname of the instance as tag “instance” and program name as tag “prog” – but NOT log name – see in gotchas bellow…
  • program is text file with description for catching metrics from log file (parameter –progs on mtail command line)

Test if mtail is running:

  • ps -ef|grep mtail
    • basic test of course
  • curl localhost:3903/metrics
    • this should show exported metrics if mtail is not running you will see error message like this “curl: (7) Failed to connect to localhost port 3903: Connection refused”

Usage:

  • be aware – restart of mtail resets all metrics !
    • but this is not so big problem since you most probably will show in graphs (probably in Grafana) rates and not absolute numbers

Gotchas:

  • Right now (2016/12) mtail has one big problem – metrics are not bind to log name. So if you try to monitor more log files you will see metrics summarized over all log files which is very unfortunate and makes mtail almost unusable

Resources: