Friday 11 January 2019

Prometheus at Home - on Mac OS

I have an old circa Mid-2010 that I am planning on using to control my Form 1 (presuming I can get resin for it) and various other things.

It struck me as a good idea to use it as a Prometheus server for my IoT research projects as part of BlueTrain Software Ltd.

Our golden trio is Prometheus, Prometheus PushGateway (for holding the job data to be scraped by Prometheus) and Grafana for graphing. Ideally putting alerts on metrics would be a good idea (we have air-con, so underneath the house suddenly dipping in temperature in the summer or heating up in the winter is a red flag a pipe has burst - for example).

Prometheus is in brew, so installing it is pretty simple.

brew install prometheus

Don't install it as a service as yet - it will install based on your login account, not on boot (which you want). You could leave it against your login account if you auto-login on boot, but that is terribly insecure, so let's not do that.

You could also use Docker for these, but Docker won't run on machine boot either, so it isn't useful here.

Prometheus Gateway isn't so easy -  you have to download the binary and copy it the right place. I grabbed the 386 darwin binary and moved it to /usr/local/bin. It runs on port 9091, so thats fine.

Now to get them both running as launchd agents. For the PushGateway, use something like this:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.prometheus.pushgateway</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/pushgateway</string>
    </array>
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>


it will just run the push gateway with the default arguments.

You need to put that in /Library/LaunchDaemons - name it org.prometheus.pushgateway.plist and then load it as sudo: sudo launchctl load org.prometheus.pushgateway.plist  (from the /Library/LaunchDaemons directory).

Trap for young players - these plist files must be owned by root:wheel if you intend to run them with sudo, so chown root:wheel X.plist your files before trying to load them.

To run prometheus, I created a /usr/local/etc/prometheus.yml file and then a modified /Library/LaunchDaemons/org.prometheus.plist file, they looked like these respectively.


# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'push-gateway'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9091']


prometheus plist file:

<?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.prometheus</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/opt/prometheus/bin/prometheus</string>
      <string>--config.file=/usr/local/etc/prometheus.yml</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <false/>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/prometheus.err.log</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/prometheus.log</string>
  </dict>
</plist>


Again, you need to load this plist file in the same was you did above.

As you can see, prometheus logs in /usr/local/var/log - so check its running.

For grafana, its pretty straight forward again:

brew install grafana

then go into the Cellar for it, in my case 5.4.2:

➜  5.4.2  pwd
/usr/local/Cellar/grafana/5.4.2
➜  5.4.2  ls
CHANGELOG.md                LICENSE.md                  README.md                   homebrew.mxcl.grafana.plist
INSTALL_RECEIPT.json        NOTICE.md                   bin                         share
➜  5.4.2  sudo cp homebrew.mxcl.grafana.plist /Library/LaunchDaemons 
➜  5.4.2  cd /Library/LaunchDaemons 
➜  LaunchDaemons  sudo chown root:wheel homebrew.mxcl.grafana.plist 
➜  LaunchDaemons  sudo launchctl load homebrew.mxcl.grafana.plist