Compare commits

...

5 Commits

7 changed files with 350 additions and 136 deletions

140
README.md
View File

@ -11,7 +11,85 @@ Can work with email, text (sms), push notifications, etc.
# Install
Git:
## Downloads
### MacOS
MacOS (darwin): [64-bit Download ](https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog)
```
curl https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog -o watchdog
```
### Windows
<details>
<summary>See download options</summary>
Windows 10: [64-bit Download](https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe)
```
powershell.exe $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe -OutFile watchdog.exe
```
Windows 7: [32-bit Download](https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe)
```
powershell.exe "(New-Object Net.WebClient).DownloadFile('https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe', 'watchdog.exe')"
```
</details>
### Linux
<details>
<summary>See download options</summary>
Linux (64-bit): [Download](https://rootprojects.org/watchdog/dist/linux/amd64/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/amd64/watchdog -o watchdog
```
Linux (32-bit): [Download](https://rootprojects.org/watchdog/dist/linux/386/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/386/watchdog -o watchdog
```
</details>
### Raspberry Pi (Linux ARM)
<details>
<summary>See download options</summary>
RPi 4 (64-bit armv8): [Download](https://rootprojects.org/watchdog/dist/linux/armv8/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv8/watchdog -o watchdog`
```
RPi 3 (armv7): [Download](https://rootprojects.org/watchdog/dist/linux/armv7/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv7/watchdog -o watchdog
```
ARMv6: [Download](https://rootprojects.org/watchdog/dist/linux/armv6/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv6/watchdog -o watchdog
```
RPi Zero (armv5): [Download](https://rootprojects.org/watchdog/dist/linux/armv5/watchdog)
```
curl https://rootprojects.org/watchdog/dist/linux/armv5/watchdog -o watchdog
```
</details>
## Git:
```bash
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
@ -21,19 +99,6 @@ pushd cmd/watchdog
go build -mod=vendor
```
Zip:
- Linux
- [watchdog-v1.1.0-linux-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-amd64.zip)
- [watchdog-v1.1.0-linux-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-386.zip)
- [watchdog-v1.1.0-linux-armv7.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv7.zip)
- [watchdog-v1.1.0-linux-armv5.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv5.zip)
- MacOS
- [watchdog-v1.1.0-darwin-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-macos/watchdog-v1.1.0-darwin-amd64.zip)
- Windows
- [watchdog-v1.1.0-windows-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-amd64.zip)
- [watchdog-v1.1.0-windows-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-386.zip)
# Usage
Mac, Linux:
@ -48,6 +113,15 @@ Windows:
watchdog.exe -c config.json
```
# Changelog
- v1.2.0
- report when sites come back up
- and more template vars
- and localization for status
- v1.1.0 support `json` request bodies (for Pushbullet)
- v1.0.0 support Twilio and Mailgun
# Getting Started
<details>
@ -78,6 +152,23 @@ Be careful of "smart quotes" and HTML entities:
- `Were Open!` is not `We're Open!`
- Neither is `We&apos;re Open!` nor `We&#39;re Open!`
Leave empty for No Content pages, such as redirects.
### `badwords`
The opposite of `keywords`.
If a literal, exact match of badwords exists as part of the response, the site is considered to be down.
Ignored if empty.
### `localizations`
Normally `{{ .Status }}` will be `"up"` or `"down"` and `{{ .Message }}` will be `"is down"` or `"came back up"`.
Localizations allow you to swap that out for something else.
I added this so that I could use "🔥🔥🔥" and "👍" for myself without imposing upon others.
### `webhooks`
This references the arbitrary `name` of a webhook in the `webhooks` array.
@ -114,7 +205,10 @@ command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-
<details>
<summary>{{ .Name }} and other template variables</summary>
`{{ .Name }}` is the only template variable right now.
- `{{ .Name }}` is the name of your site.
- `{{ .Message }}` is either `went down` or `came back up`.
- `{{ .Status }}` is either `up` or `down`.
- `{{ .Watchdog }}` is the name of your watchdog (useful if you have multiple).
It refers to the name of the watch, which is "Example Site" in the sample config below.
@ -234,11 +328,13 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
```json
{
"watchdog": "Monitor A",
"watches": [
{
"name": "Example Site",
"url": "https://example.com/",
"keywords": "My Site",
"badwords": "Could not connect to database.",
"webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
"recover_script": "systemctl restart example-site"
}
@ -258,8 +354,8 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
"form": {
"from": "Watchdog <watchdog@my.example.com>",
"to": "jon.doe@gmail.com",
"subject": "{{ .Name }} is down.",
"text": "The system is down. Check up on {{ .Name }} ASAP."
"subject": "[{{ .Watchdog }}] {{ .Name }} {{ .Message }}.",
"text": "{{ .Name }} {{ .Message }}. Reported by {{ .Watchdog }}."
}
},
{
@ -271,8 +367,8 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
"User-Agent": "Watchdog/1.0"
},
"json": {
"body": "The system is down. Check up on {{ .Name }} ASAP.",
"title": "{{ .Name }} is down.",
"body": "The system {{ .Message }}. Check up on {{ .Name }} ASAP.",
"title": "{{ .Name }} {{ .Message }}.",
"type": "note"
}
},
@ -293,7 +389,11 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
"Body": "[{{ .Name }}] The system is down. The system is down."
}
}
]
],
"localizations": {
"up": "👍",
"down": "🔥🔥🔥"
}
}
```

48
build-all.sh Normal file
View File

@ -0,0 +1,48 @@
#GOOS=windows GOARCH=amd64 go install
#go tool dist list
# TODO move this into tools/build.go
export CGO_ENABLED=0
exe=watchdog
distpre=../..
gocmd=.
echo ""
go generate -mod=vendor ./...
pushd cmd/${exe}
echo ""
echo "Windows amd64"
#GOOS=windows GOARCH=amd64 go build -mod=vendor -o ${distpre}/dist/windows/amd64/${exe}.exe -ldflags "-H=windowsgui" $gocmd
#GOOS=windows GOARCH=amd64 go build -mod=vendor -o ${distpre}/dist/windows/amd64/${exe}.debug.exe
GOOS=windows GOARCH=amd64 go build -mod=vendor -o ${distpre}/dist/windows/amd64/${exe}.exe
echo "Windows 386"
#GOOS=windows GOARCH=386 go build -mod=vendor -o ${distpre}/dist/windows/386/${exe}.exe -ldflags "-H=windowsgui" $gocmd
#GOOS=windows GOARCH=386 go build -mod=vendor -o ${distpre}/dist/windows/386/${exe}.debug.exe
GOOS=windows GOARCH=386 go build -mod=vendor -o ${distpre}/dist/windows/386/${exe}.exe
echo ""
echo "Darwin (macOS) amd64"
GOOS=darwin GOARCH=amd64 go build -mod=vendor -o ${distpre}/dist/darwin/amd64/${exe} $gocmd
echo ""
echo "Linux amd64"
GOOS=linux GOARCH=amd64 go build -mod=vendor -o ${distpre}/dist/linux/amd64/${exe} $gocmd
echo "Linux 386"
GOOS=linux GOARCH=386 go build -mod=vendor -o ${distpre}/dist/linux/386/${exe} $gocmd
echo ""
echo "RPi 4 (64-bit) ARMv8"
GOOS=linux GOARCH=arm64 go build -mod=vendor -o ${distpre}/dist/linux/armv8/${exe} $gocmd
echo "RPi 3 B+ ARMv7"
GOOS=linux GOARCH=arm GOARM=7 go build -mod=vendor -o ${distpre}/dist/linux/armv7/${exe} $gocmd
echo "ARMv6"
GOOS=linux GOARCH=arm GOARM=6 go build -mod=vendor -o ${distpre}/dist/linux/armv6/${exe} $gocmd
echo "RPi Zero ARMv5"
GOOS=linux GOARCH=arm GOARM=5 go build -mod=vendor -o ${distpre}/dist/linux/armv5/${exe} $gocmd
echo ""
popd
rsync -av ./dist/ ubuntu@rootprojects.org:/srv/www/rootprojects.org/$exe/dist/
# https://rootprojects.org/serviceman/dist/windows/amd64/serviceman.exe

View File

@ -1,45 +0,0 @@
#!/usr/bin/env bash
export CGO_ENABLED=0
#GOOS=windows GOARCH=amd64 go install
go tool dist list
gocmd=watchdog.go
golib=""
echo ""
echo ""
echo "Windows amd64"
GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/watchdog.exe $gocmd $golib
echo "Windows 386"
GOOS=windows GOARCH=386 go build -o dist/windows-386/watchdog.exe $gocmd $golib
echo ""
echo "Darwin (macOS) amd64"
GOOS=darwin GOARCH=amd64 go build -o dist/darwin-amd64/watchdog $gocmd $golib
echo ""
echo "Linux amd64"
GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/watchdog $gocmd $golib
echo "Linux 386"
echo ""
GOOS=linux GOARCH=386 go build -o dist/linux-386/watchdog $gocmd $golib
echo "RPi 3 B+ ARMv7"
GOOS=linux GOARCH=arm GOARM=7 go build -o dist/linux-armv7/watchdog $gocmd $golib
echo "RPi Zero ARMv5"
GOOS=linux GOARCH=arm GOARM=5 go build -o dist/linux-armv5/watchdog $gocmd $golib
my_ver=$(git describe --tags)
pushd dist
ls -d *-* | while read my_dist
do
if [ -d "$my_dist" ]; then
#tar -czvf watchdog-$my_ver-$my_dist.tar.gz $my_dist
zip -r watchdog-$my_ver-$my_dist.zip $my_dist
fi
done
popd
echo ""
echo ""

View File

@ -11,7 +11,7 @@ import (
"os"
"strings"
watchdog "git.rootprojects.org/root/watchdog.go"
watchdog "git.rootprojects.org/root/go-watchdog"
)
var GitRev, GitVersion, GitTimestamp string
@ -83,13 +83,16 @@ func main() {
logQueue <- fmt.Sprintf("Watching '%s'", c.Name)
go func(c watchdog.ConfigWatch) {
d := watchdog.New(&watchdog.Dog{
Name: c.Name,
CheckURL: c.URL,
Keywords: c.Keywords,
Recover: c.RecoverScript,
Webhooks: c.Webhooks,
AllWebhooks: allWebhooks,
Logger: logQueue,
Watchdog: config.Watchdog,
Name: c.Name,
CheckURL: c.URL,
Keywords: c.Keywords,
Badwords: c.Badwords,
Localizations: config.Localizations,
Recover: c.RecoverScript,
Webhooks: c.Webhooks,
AllWebhooks: allWebhooks,
Logger: logQueue,
})
d.Watch()
}(config.Watches[i])

2
doc.go
View File

@ -5,5 +5,5 @@
// The git tag version describes the state of the binary,
// not the state of the library. The API is not yet stable.
//
// See https://git.rootproject.org/root/watchdog.go for pre-built binaries.
// See https://git.rootproject.org/root/go-watchdog for pre-built binaries.
package watchdog

2
go.mod
View File

@ -1,4 +1,4 @@
module git.rootprojects.org/root/watchdog.go
module git.rootprojects.org/root/go-watchdog
go 1.12

View File

@ -14,24 +14,56 @@ import (
"time"
)
type Status int
const (
StatusDown Status = iota
StatusUp
)
func (s Status) String() string {
// ... just wishing Go had enums like Rust...
switch s {
case StatusUp:
return "up"
case StatusDown:
return "down"
default:
return "[[internal error]]"
}
}
const (
MessageDown = "went down"
MessageUp = "came back up"
MessageHiccup = "hiccupped"
)
type Dog struct {
Name string
CheckURL string
Keywords string
Recover string
Webhooks []string
AllWebhooks map[string]Webhook
Logger chan string
error error
failures int
passes int
lastFailed time.Time
lastPassed time.Time
lastNotified time.Time
Watchdog string
Name string
CheckURL string
Keywords string
Badwords string
Localizations map[string]string
Recover string
Webhooks []string
AllWebhooks map[string]Webhook
Logger chan string
status Status
changed bool
error error
//failures int
//passes int
//lastFailed time.Time
//lastPassed time.Time
//lastNotified time.Time
}
func New(d *Dog) *Dog {
d.lastPassed = time.Now().Add(-5 * time.Minute)
//d.lastPassed = time.Now().Add(-5 * time.Minute)
d.status = StatusUp
d.changed = false
return d
}
@ -44,64 +76,87 @@ func (d *Dog) Watch() {
}
}
// Now that I've added the ability to notify when a server is back up
// this definitely needs some refactoring. It's bad now.
func (d *Dog) watch() {
d.Logger <- fmt.Sprintf("Check: '%s'", d.Name)
err := d.check()
// This may be up or down
err := d.hardcheck()
if nil == err {
d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
// if it's down, coming up, notify
if d.changed {
d.notify(MessageUp)
}
return
}
time.Sleep(time.Duration(2) * time.Second)
err2 := d.check()
if nil != err2 {
d.Logger <- fmt.Sprintf("Down: '%s': %s", d.Name, err2)
} else {
d.Logger <- fmt.Sprintf("Hiccup: '%s': %s", d.Name, err)
return
// If being down is a change, check to see if it's just a hiccup
if d.changed {
time.Sleep(time.Duration(5) * time.Second)
err2 := d.softcheck()
if nil != err2 {
// it's really down
d.Logger <- fmt.Sprintf("Down: '%s': %s", d.Name, err2)
} else {
// it's not really down, so reset the change info
d.changed = false
d.status = StatusUp
// and notify of the hiccup
d.Logger <- fmt.Sprintf("Hiccup: '%s': %s", d.Name, err)
d.notify(MessageHiccup)
return
}
}
failure := false
// TODO what if the server is flip-flopping rapidly?
// how to rate limit?
// "{{ .Server }} is on cooldown for 30 minutes"
// * We've had success since the last notification
// * It's been at least 5 minutes since the last notification
//fiveMinutesAgo := time.Now().Add(-5 * time.Minute)
//if d.lastPassed.After(d.lastNotified) && d.lastNotified.Before(fiveMinutesAgo) {
//}
t := 10
for {
// try to recover, then backoff exponentially
d.recover()
time.Sleep(time.Duration(t) * time.Second)
// backoff
t *= 2
err := d.check()
if nil != err {
d.Logger <- fmt.Sprintf("Unrecoverable: '%s': %s", d.Name, err)
failure = true
} else {
failure = false
if t > 120 {
t = 120
}
// We should notify if
// * We've had success since the last notification
// * It's been at least 5 minutes since the last notification
fiveMinutesAgo := time.Now().Add(-5 * time.Minute)
if d.lastPassed.After(d.lastNotified) && d.lastNotified.Before(fiveMinutesAgo) {
d.notify(failure)
}
if !failure || d.failures >= 5 {
// go back to the main 5-minute loop
err := d.softcheck()
if nil != err {
// this is down, and we know it's down
d.status = StatusDown
d.Logger <- fmt.Sprintf("Unrecoverable: '%s': %s", d.Name, err)
if d.changed {
d.changed = false
d.notify(MessageDown)
}
} else {
// it came back up
d.status = StatusUp
d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
if d.changed {
// and the downtime was short - just a recovery
d.notify(MessageHiccup)
} else {
// and the downtime was some time
d.notify(MessageUp)
}
d.changed = false
break
}
}
}
func (d *Dog) check() error {
var err error
defer func() {
if nil != err {
d.failures += 1
d.lastFailed = time.Now()
} else {
d.lastPassed = time.Now()
d.passes += 1
}
}()
func (d *Dog) softcheck() error {
client := NewHTTPClient()
response, err := client.Get(d.CheckURL)
if nil != err {
@ -115,18 +170,52 @@ func (d *Dog) check() error {
return err
}
// Note: empty matches empty as true, so this works for checking redirects
if !bytes.Contains(b, []byte(d.Keywords)) {
err = fmt.Errorf("Down: '%s' Not Found for '%s'", d.Keywords, d.Name)
d.Logger <- fmt.Sprintf("%s", err)
d.error = err
return err
} else {
d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
}
if "" != d.Badwords {
if bytes.Contains(b, []byte(d.Badwords)) {
err = fmt.Errorf("Down: '%s' Found for '%s'", d.Badwords, d.Name)
d.Logger <- fmt.Sprintf("%s", err)
d.error = err
return err
}
}
return nil
}
func (d *Dog) hardcheck() error {
previousStatus := d.status
err := d.softcheck()
// Are we up, or down?
if nil != err {
d.status = StatusDown
//d.failures += 1
//d.lastFailed = time.Now()
} else {
d.status = StatusUp
//d.lastPassed = time.Now()
//d.passes += 1
}
// Has that changed?
if previousStatus != d.status {
d.changed = true
} else {
d.changed = false
}
return err
}
func (d *Dog) recover() {
if "" == d.Recover {
return
@ -154,9 +243,9 @@ func (d *Dog) recover() {
}
}
func (d *Dog) notify(hardFail bool) {
d.Logger <- fmt.Sprintf("Notifying the authorities of %s's failure", d.Name)
d.lastNotified = time.Now()
func (d *Dog) notify(msg string) {
d.Logger <- fmt.Sprintf("Notifying the authorities of %s's status change", d.Name)
//d.lastNotified = time.Now()
for i := range d.Webhooks {
name := d.Webhooks[i]
@ -172,11 +261,11 @@ func (d *Dog) notify(hardFail bool) {
continue
}
d.notifyOne(h, hardFail)
d.notifyOne(h, msg)
}
}
func (d *Dog) notifyOne(h Webhook, hardFail bool) {
func (d *Dog) notifyOne(h Webhook, msg string) {
// TODO do this in main on config init
if "" == h.Method {
h.Method = "POST"
@ -191,7 +280,10 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
v := h.Form[k]
// because `{{` gets urlencoded
//k = strings.Replace(k, "{{ .Name }}", d.Name, -1)
v = strings.Replace(v, "{{ .Watchdog }}", d.Watchdog, -1)
v = strings.Replace(v, "{{ .Name }}", d.Name, -1)
v = strings.Replace(v, "{{ .Status }}", d.localize(d.status.String()), -1)
v = strings.Replace(v, "{{ .Message }}", d.localize(msg), -1)
d.Logger <- fmt.Sprintf("[HEADER] %s: %s", k, v)
form.Set(k, v)
}
@ -203,7 +295,12 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
return
}
// `{{` should be left alone
body = strings.NewReader(strings.Replace(string(bodyBuf), "{{ .Name }}", d.Name, -1))
v := string(bodyBuf)
v = strings.Replace(v, "{{ .Watchdog }}", d.Watchdog, -1)
v = strings.Replace(v, "{{ .Name }}", d.Name, -1)
v = strings.Replace(v, "{{ .Status }}", d.localize(d.status.String()), -1)
v = strings.Replace(v, "{{ .Message }}", d.localize(msg), -1)
body = strings.NewReader(v)
}
client := NewHTTPClient()
@ -260,16 +357,27 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
// TODO some sort of way to determine if data is successful (keywords)
d.Logger <- fmt.Sprintf("[Notify] Success? %#v", data)
}
func (d *Dog) localize(msg string) string {
for k := range d.Localizations {
if k == msg {
return d.Localizations[k]
}
}
return msg
}
type Config struct {
Watches []ConfigWatch `json:"watches"`
Webhooks []Webhook `json:"webhooks"`
Watchdog string `json:"watchdog"`
Watches []ConfigWatch `json:"watches"`
Webhooks []Webhook `json:"webhooks"`
Localizations map[string]string `json:"localizations"`
}
type ConfigWatch struct {
Name string `json:"name"`
URL string `json:"url"`
Keywords string `json:"keywords"`
Badwords string `json:"badwords"`
Webhooks []string `json:"webhooks"`
RecoverScript string `json:"recover_script"`
}