Watchdog

Get notified when sites go down.

Watchdog: Webhooks for all the times when something doesn't go right.

For every url listed in the config, check if the specified keywords are found on the page every 5 minutes. Whenever they aren't found execute the recover_script and make a request to each of the associated webhooks.

Can work with email, text (sms), push notifications, etc.

Install

Downloads

MacOS

MacOS (darwin): 64-bit Download

curl https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog -o watchdog

Windows

See download options

Windows 10: [64-bit Download](https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe)

powershell.exe $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe -OutFile watchdog.exe

Windows 7: 32-bit Download

powershell.exe "(New-Object Net.WebClient).DownloadFile('https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe', 'watchdog.exe')"

Linux

See download options

Linux (64-bit): Download

curl https://rootprojects.org/watchdog/dist/linux/amd64/watchdog -o watchdog

Linux (32-bit): Download

curl https://rootprojects.org/watchdog/dist/linux/386/watchdog -o watchdog

Raspberry Pi (Linux ARM)

See download options

RPi 4 (64-bit armv8): Download

curl https://rootprojects.org/watchdog/dist/linux/armv8/watchdog -o watchdog`

RPi 3 (armv7): Download

curl https://rootprojects.org/watchdog/dist/linux/armv7/watchdog -o watchdog

ARMv6: Download

curl https://rootprojects.org/watchdog/dist/linux/armv6/watchdog -o watchdog

RPi Zero (armv5): Download

curl https://rootprojects.org/watchdog/dist/linux/armv5/watchdog -o watchdog

Git:

git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
pushd watchdog.go/
go generate -mod=vendor ./...
pushd cmd/watchdog
go build -mod=vendor

Usage

Mac, Linux:

./watchdog -c config.json

Windows:

watchdog.exe -c config.json

Changelog

• v1.2.0
  • report when sites come back up
  • and more template vars
  • and localization for status
• v1.1.0 support json request bodies (for Pushbullet)
• v1.0.0 support Twilio and Mailgun

Getting Started

How do I configure what to watch?

name

This is an arbitrary name which can be used as {{ .Name }} in the template strings.

In most cases I would probably set mine to the literal domain name (such as git.rootprojects.org), but you can do whatever makes sense to you.

If you use it in a template, you probably shouldn't allow it to be arbitrary user input.

url

This is the page the watchdog will check for the exact match of keywords you set.

It's a get request. I actually want to be regularly check that contact forms are working, so I may very well add more functionality to this in the future.

keywords

The url will be checked for a literal, exact match of keywords.

Be careful of "smart quotes" and HTML entities:

• We’re Open! is not We're Open!
• Neither is We're Open! nor We're Open!

Leave empty for No Content pages, such as redirects.

badwords

The opposite of keywords.

If a literal, exact match of badwords exists as part of the response, the site is considered to be down.

Ignored if empty.

localizations

Normally {{ .Status }} will be "up" or "down" and {{ .Message }} will be "is down" or "came back up". Localizations allow you to swap that out for something else.

I added this so that I could use "🔥🔥🔥" and "👍" for myself without imposing upon others.

webhooks

This references the arbitrary name of a webhook in the webhooks array.

These webhooks will be run in order and the next will still be run even when the previous fails.

recover_script

The full contents of this file will be passed to bash as if it were a file.

You can run a single line, such as ssh watchdog@my.example.com 'systemctl restart foo.service' or an entire script.

Pro-Tip™

Don't forget that you can add single-use ssh keys to run specific commands on a remote system.

On the system that runs the watchdog you would create a new, single-use key, like this:

ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
cat ~/.ssh/restart-service-foo.pub

On the system that runs the service being watched you would add that key restricted to the single command:

/root/.ssh/authorized_keys:

command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog

{{ .Name }} and other template variables

• {{ .Name }} is the name of your site.
• {{ .Message }} is either went down or came back up.
• {{ .Status }} is either up or down.
• {{ .Watchdog }} is the name of your watchdog (useful if you have multiple).

It refers to the name of the watch, which is "Example Site" in the sample config below.

How to use with Mailgun

my_mailgun

Replace my_mailgun with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference. For example, you may have different mailgun configurations for different domains.

my.example.com

Replace both instances of my.example.com with your domain in mailgun.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Replace the HTTP Basic Auth "password" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your API token.

api stays, it is the expected "username".

to, from, subject, & text

Hopefully it's obivous enough that you should send your alerts to yourself, not jon.doe@gmail.com but, just in case: change that to your email.

The from address can be any domain in your mailgun account.

subject is the plain-text subject and text must be plain-text (not html) email contents.

How to use with Pushbullet

Pushbullet is a push notification service that I found pretty easy to work with. I logged in with my iPhone through facebook, as well as in a web browser, grabbed the API token, and I was able to test the documentation all in just a few minutes.

my_pushbullet

Replace my_pushbullet with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference. For example, you may have different Pushbullet configurations for different domains.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Replace xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx in the HTTP Access-Token header with your API token.

You'll find this on the Pushbullet dashboard under Settings, Account, Access Tokens, and you'll have to click to create it:

• https://www.pushbullet.com/#settings/account

The example was taken from the Pushbullet API documentation:

• https://docs.pushbullet.com/#create-push

How to use with Twilio

my_twilio

Replace my_twilio with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference. For example, you may have different twilio configurations for different domains.

AC00000000000000000000000000000000

This is a placeholder for your sid, also called "Account SID".

Replace both instances of AC00000000000000000000000000000000 with the sid shown in your twilio dashboard.

This is used both in the url as well as in the auth.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Replace the HTTP Basic Auth "password" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your API token.

You'll find this on the twilio dashboard with your "Account SID", but you'll have to click to reveal it.

To, From, & Body

All phone numbers should have the country code prefix (+1 for USA) attached.

To is the number that you want to send the text to.

From must be a number listed in your twilio account.

Body is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".

How to use with Nexmo, Mailjet, and other webhook-enabled services

See the examples of Mailgun, Pushbullet, Twilio.

Look for "curl" in the documentation of the service that you're using. It should be fairly easy to just look at the headers that are being set and repeat.

Sample Config

You can set notifications for any service that supports HTTPS webhooks.

The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from their curl documentation.

{
  "watchdog": "Monitor A",
  "watches": [
    {
      "name": "Example Site",
      "url": "https://example.com/",
      "keywords": "My Site",
      "badwords": "Could not connect to database.",
      "webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
      "recover_script": "systemctl restart example-site"
    }
  ],
  "webhooks": [
    {
      "name": "my_mailgun",
      "method": "POST",
      "url": "https://api.mailgun.net/v3/my.example.com/messages",
      "auth": {
        "username": "api",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "headers": {
        "User-Agent": "Watchdog/1.0"
      },
      "form": {
        "from": "Watchdog <watchdog@my.example.com>",
        "to": "jon.doe@gmail.com",
        "subject": "[{{ .Watchdog }}] {{ .Name }} {{ .Message }}.",
        "text": "{{ .Name }} {{ .Message }}. Reported by {{ .Watchdog }}."
      }
    },
    {
      "name": "my_pushbullet",
      "method": "POST",
      "url": "https://api.pushbullet.com/v2/pushes",
      "headers": {
        "Access-Token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "User-Agent": "Watchdog/1.0"
      },
      "json": {
        "body": "The system {{ .Message }}. Check up on {{ .Name }} ASAP.",
        "title": "{{ .Name }} {{ .Message }}.",
        "type": "note"
      }
    },
    {
      "name": "my_twilio",
      "method": "POST",
      "url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
      "auth": {
        "username": "AC00000000000000000000000000000000",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "headers": {
        "User-Agent": "Watchdog/1.0"
      },
      "form": {
        "To": "+1 801 555 1234",
        "From": "+1 800 555 4321",
        "Body": "[{{ .Name }}] The system is down. The system is down."
      }
    }
  ],
  "localizations": {
    "up": "👍",
    "down": "🔥🔥🔥"
  }
}