AJ ONeal
1e9f95295d
|
||
|---|---|---|
| cmd | ||
| tools | ||
| vendor | ||
| .gitignore | ||
| README.md | ||
| build.sh | ||
| doc.go | ||
| go.mod | ||
| go.sum | ||
| watchdog.go | ||
README.md
Watchdog
Get notified when sites go down.
Watchdog: Webhooks for all the times when something doesn't go right.
For every url listed in the config, check if the specified keywords are found on the page every 5 minutes.
Whenever they aren't found execute the recover_script and make a request to each of the associated webhooks.
Can work with email, text (sms), push notifications, etc.
Install
Git:
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
pushd watchdog.go/
go generate -mod=vendor ./...
pushd cmd/watchdog
go build -mod=vendor
Zip:
- Linux
- MacOS
- Windows
Usage
Mac, Linux:
./watchdog -c config.json
Windows:
watchdog.exe -c config.json
Getting Started
How do I configure what to watch?
name
This is an arbitrary name which can be used as {{ .Name }} in the template strings.
In most cases I would probably set mine to the literal domain name (such as git.rootprojects.org),
but you can do whatever makes sense to you.
If you use it in a template, you probably shouldn't allow it to be arbitrary user input.
url
This is the page the watchdog will check for the exact match of keywords you set.
It's a get request. I actually want to be regularly check that contact forms are working, so I may very well add more functionality to this in the future.
keywords
The url will be checked for a literal, exact match of keywords.
Be careful of "smart quotes" and HTML entities:
We’re Open!is notWe're Open!- Neither is
We're Open!norWe're Open!
webhooks
This references the arbitrary name of a webhook in the webhooks array.
These webhooks will be run in order and the next will still be run even when the previous fails.
recover_script
The full contents of this file will be passed to bash as if it were a file.
You can run a single line, such as ssh watchdog@my.example.com 'systemctl restart foo.service'
or an entire script.
Pro-Tip™
Don't forget that you can add single-use ssh keys to run specific commands on a remote system.
On the system that runs the watchdog you would create a new, single-use key, like this:
ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
cat ~/.ssh/restart-service-foo.pub
On the system that runs the service being watched you would add that key restricted to the single command:
/root/.ssh/authorized_keys:
command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog
{{ .Name }} and other template variables
{{ .Name }} is the only template variable right now.
It refers to the name of the watch, which is "Example Site" in the sample config below.
How to use with Mailgun
my_mailgun
Replace my_mailgun with whatever name you like, or leave it the same.
It's an arbitrary name for you to reference. For example, you may have different mailgun configurations for different domains.
my.example.com
Replace both instances of my.example.com with your domain in mailgun.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Replace the HTTP Basic Auth "password" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
with your API token.
api stays, it is the expected "username".
to, from, subject, & text
Hopefully it's obivous enough that you should send your alerts to yourself,
not jon.doe@gmail.com but, just in case: change that to your email.
The from address can be any domain in your mailgun account.
subject is the plain-text subject and text must be plain-text (not html) email contents.
How to use with Pushbullet
Pushbullet is a push notification service that I found pretty easy to work with. I logged in with my iPhone through facebook, as well as in a web browser, grabbed the API token, and I was able to test the documentation all in just a few minutes.
my_pushbullet
Replace my_pushbullet with whatever name you like, or leave it the same.
It's an arbitrary name for you to reference. For example, you may have different Pushbullet configurations for different domains.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Replace xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx in the HTTP Access-Token header
with your API token.
You'll find this on the Pushbullet dashboard under Settings, Account, Access Tokens, and you'll have to click to create it:
The example was taken from the Pushbullet API documentation:
How to use with Twilio
my_twilio
Replace my_twilio with whatever name you like, or leave it the same.
It's an arbitrary name for you to reference. For example, you may have different twilio configurations for different domains.
AC00000000000000000000000000000000
This is a placeholder for your sid, also called "Account SID".
Replace both instances of AC00000000000000000000000000000000 with the sid shown in your twilio dashboard.
This is used both in the url as well as in the auth.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Replace the HTTP Basic Auth "password" xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
with your API token.
You'll find this on the twilio dashboard with your "Account SID", but you'll have to click to reveal it.
To, From, & Body
All phone numbers should have the country code prefix (+1 for USA) attached.
To is the number that you want to send the text to.
From must be a number listed in your twilio account.
Body is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".
How to use with Nexmo, Mailjet, and other webhook-enabled services
See the examples of Mailgun, Pushbullet, Twilio.
Look for "curl" in the documentation of the service that you're using. It should be fairly easy to just look at the headers that are being set and repeat.
Sample Config
You can set notifications for any service that supports HTTPS webhooks.
The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from their curl documentation.
{
"watches": [
{
"name": "Example Site",
"url": "https://example.com/",
"keywords": "My Site",
"webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
"recover_script": "systemctl restart example-site"
}
],
"webhooks": [
{
"name": "my_mailgun",
"method": "POST",
"url": "https://api.mailgun.net/v3/my.example.com/messages",
"auth": {
"username": "api",
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"headers": {
"User-Agent": "Watchdog/1.0"
},
"form": {
"from": "Watchdog <watchdog@my.example.com>",
"to": "jon.doe@gmail.com",
"subject": "{{ .Name }} is down.",
"text": "The system is down. Check up on {{ .Name }} ASAP."
}
},
{
"name": "my_pushbullet",
"method": "POST",
"url": "https://api.pushbullet.com/v2/pushes",
"headers": {
"Access-Token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"User-Agent": "Watchdog/1.0"
},
"json": {
"body": "The system is down. Check up on {{ .Name }} ASAP.",
"title": "{{ .Name }} is down.",
"type": "note"
}
},
{
"name": "my_twilio",
"method": "POST",
"url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
"auth": {
"username": "AC00000000000000000000000000000000",
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"headers": {
"User-Agent": "Watchdog/1.0"
},
"form": {
"To": "+1 801 555 1234",
"From": "+1 800 555 4321",
"Body": "[{{ .Name }}] The system is down. The system is down."
}
}
]
}