v1.2.0: back up notifications and other features

2019-07-04 18:23:51 -06:00 · 2019-07-04 18:23:51 -06:00 · 8ba8dd03dc
parent 94c0dfa2a0
commit 8ba8dd03dc
7 changed files with 290 additions and 110 deletions
--- a/README.md
+++ b/README.md
@ -11,7 +11,85 @@ Can work with email, text (sms), push notifications, etc.

 # Install

-Git:
+## Downloads
+
+### MacOS
+
+MacOS (darwin): [64-bit Download ](https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog -o watchdog
+```
+
+### Windows
+
+<details>
+<summary>See download options</summary>
+Windows 10: [64-bit Download](https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe)
+
+```
+powershell.exe $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe -OutFile watchdog.exe
+```
+
+Windows 7: [32-bit Download](https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe)
+
+```
+powershell.exe "(New-Object Net.WebClient).DownloadFile('https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe', 'watchdog.exe')"
+```
+
+</details>
+
+### Linux
+
+<details>
+<summary>See download options</summary>
+
+Linux (64-bit): [Download](https://rootprojects.org/watchdog/dist/linux/amd64/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/linux/amd64/watchdog -o watchdog
+```
+
+Linux (32-bit): [Download](https://rootprojects.org/watchdog/dist/linux/386/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/linux/386/watchdog -o watchdog
+```
+
+</details>
+
+### Raspberry Pi (Linux ARM)
+
+<details>
+<summary>See download options</summary>
+
+RPi 4 (64-bit armv8): [Download](https://rootprojects.org/watchdog/dist/linux/armv8/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/linux/armv8/watchdog -o watchdog`
+```
+
+RPi 3 (armv7): [Download](https://rootprojects.org/watchdog/dist/linux/armv7/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/linux/armv7/watchdog -o watchdog
+```
+
+ARMv6: [Download](https://rootprojects.org/watchdog/dist/linux/armv6/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/linux/armv6/watchdog -o watchdog
+```
+
+RPi Zero (armv5): [Download](https://rootprojects.org/watchdog/dist/linux/armv5/watchdog)
+
+```
+curl https://rootprojects.org/watchdog/dist/linux/armv5/watchdog -o watchdog
+```
+
+</details>
+
+## Git:

 ```bash
 git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
@ -21,19 +99,6 @@ pushd cmd/watchdog
 go build -mod=vendor
 ```

-Zip:
-
- Linux
-  - [watchdog-v1.1.0-linux-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-amd64.zip)
-  - [watchdog-v1.1.0-linux-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-386.zip)
-  - [watchdog-v1.1.0-linux-armv7.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv7.zip)
-  - [watchdog-v1.1.0-linux-armv5.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv5.zip)
- MacOS
-  - [watchdog-v1.1.0-darwin-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-macos/watchdog-v1.1.0-darwin-amd64.zip)
- Windows
-  - [watchdog-v1.1.0-windows-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-amd64.zip)
-  - [watchdog-v1.1.0-windows-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-386.zip)
-
 # Usage

 Mac, Linux:
@ -48,6 +113,15 @@ Windows:
 watchdog.exe -c config.json
 ```

+# Changelog
+
+- v1.2.0
+  - report when sites come back up
+  - and more template vars
+  - and localization for status
+- v1.1.0 support `json` request bodies (for Pushbullet)
+- v1.0.0 support Twilio and Mailgun
+
 # Getting Started

 <details>
@ -78,6 +152,23 @@ Be careful of "smart quotes" and HTML entities:
 - `We’re Open!` is not `We're Open!`
 - Neither is `We&apos;re Open!` nor `We&#39;re Open!`

+Leave empty for No Content pages, such as redirects.
+
+### `badwords`
+
+The opposite of `keywords`.
+
+If a literal, exact match of badwords exists as part of the response, the site is considered to be down.
+
+Ignored if empty.
+
+### `localizations`
+
+Normally `{{ .Status }}` will be `"up"` or `"down"` and `{{ .Message }}` will be `"is down"` or `"came back up"`.
+Localizations allow you to swap that out for something else.
+
+I added this so that I could use "🔥🔥🔥" and "👍" for myself without imposing upon others.
+
 ### `webhooks`

 This references the arbitrary `name` of a webhook in the `webhooks` array.
@ -114,7 +205,10 @@ command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-
 <details>
  <summary>{{ .Name }} and other template variables</summary>

-`{{ .Name }}` is the only template variable right now.
+`{{ .Name }}` is the name of your site.
+`{{ .Message }}` is either `went down` or `came back up`.
+`{{ .Status }}` is either `up` or `down`.
+`{{ .Watchdog }}` is the name of your watchdog (useful if you have multiple).

 It refers to the name of the watch, which is "Example Site" in the sample config below.

@ -234,11 +328,13 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from

 ```json
 {
+  "watchdog": "Monitor A",
  "watches": [
    {
      "name": "Example Site",
      "url": "https://example.com/",
      "keywords": "My Site",
+      "badwords": "Could not connect to database.",
      "webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
      "recover_script": "systemctl restart example-site"
    }
@ -258,8 +354,8 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
      "form": {
        "from": "Watchdog <watchdog@my.example.com>",
        "to": "jon.doe@gmail.com",
-        "subject": "{{ .Name }} is down.",
-        "text": "The system is down. Check up on {{ .Name }} ASAP."
+        "subject": "[{{ .Watchdog }}] {{ .Name }} {{ .Message }}.",
+        "text": "{{ .Name }} {{ .Message }}. Reported by {{ .Watchdog }}."
      }
    },
    {
@ -271,8 +367,8 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
        "User-Agent": "Watchdog/1.0"
      },
      "json": {
-        "body": "The system is down. Check up on {{ .Name }} ASAP.",
-        "title": "{{ .Name }} is down.",
+        "body": "The system {{ .Message }}. Check up on {{ .Name }} ASAP.",
+        "title": "{{ .Name }} {{ .Message }}.",
        "type": "note"
      }
    },
@ -293,7 +389,11 @@ The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from
        "Body": "[{{ .Name }}] The system is down. The system is down."
      }
    }
-  ]
+  ],
+  "localizations": {
+    "up": "👍",
+    "down": "🔥🔥🔥"
+  }
 }
 ```

--- a/build-all.sh
+++ b/build-all.sh
@ -0,0 +1,45 @@
+#GOOS=windows GOARCH=amd64 go install
+#go tool dist list
+
+# TODO move this into tools/build.go
+
+export CGO_ENABLED=0
+exe=watchdog
+gocmd=.
+
+echo ""
+go generate -mod=vendor ./...
+
+echo ""
+echo "Windows amd64"
+#GOOS=windows GOARCH=amd64 go build -mod=vendor -o dist/windows/amd64/${exe}.exe -ldflags "-H=windowsgui" $gocmd
+#GOOS=windows GOARCH=amd64 go build -mod=vendor -o dist/windows/amd64/${exe}.debug.exe
+GOOS=windows GOARCH=amd64 go build -mod=vendor -o dist/windows/amd64/${exe}.exe
+echo "Windows 386"
+#GOOS=windows GOARCH=386 go build -mod=vendor -o dist/windows/386/${exe}.exe -ldflags "-H=windowsgui" $gocmd
+#GOOS=windows GOARCH=386 go build -mod=vendor -o dist/windows/386/${exe}.debug.exe
+GOOS=windows GOARCH=386 go build -mod=vendor -o dist/windows/386/${exe}.exe
+
+echo ""
+echo "Darwin (macOS) amd64"
+GOOS=darwin GOARCH=amd64 go build -mod=vendor -o dist/darwin/amd64/${exe} $gocmd
+
+echo ""
+echo "Linux amd64"
+GOOS=linux GOARCH=amd64 go build -mod=vendor -o dist/linux/amd64/${exe} $gocmd
+echo "Linux 386"
+GOOS=linux GOARCH=386 go build -mod=vendor -o dist/linux/386/${exe} $gocmd
+
+echo ""
+echo "RPi 4 (64-bit) ARMv8"
+GOOS=linux GOARCH=arm64 go build -mod=vendor -o dist/linux/armv8/${exe} $gocmd
+echo "RPi 3 B+ ARMv7"
+GOOS=linux GOARCH=arm GOARM=7 go build -mod=vendor -o dist/linux/armv7/${exe} $gocmd
+echo "ARMv6"
+GOOS=linux GOARCH=arm GOARM=6 go build -mod=vendor -o dist/linux/armv6/${exe} $gocmd
+echo "RPi Zero ARMv5"
+GOOS=linux GOARCH=arm GOARM=5 go build -mod=vendor -o dist/linux/armv5/${exe} $gocmd
+
+echo ""
+rsync -av ./dist/ ubuntu@rootprojects.org:/srv/www/rootprojects.org/$exe/dist/
+# https://rootprojects.org/serviceman/dist/windows/amd64/serviceman.exe
--- a/build.sh
+++ b/build.sh
@ -1,45 +0,0 @@
-#!/usr/bin/env bash
-
-export CGO_ENABLED=0
-#GOOS=windows GOARCH=amd64 go install
-go tool dist list
-
-gocmd=watchdog.go
-golib=""
-echo ""
-
-echo ""
-echo "Windows amd64"
-GOOS=windows GOARCH=amd64 go build -o dist/windows-amd64/watchdog.exe $gocmd $golib
-echo "Windows 386"
-GOOS=windows GOARCH=386 go build -o dist/windows-386/watchdog.exe $gocmd $golib
-
-echo ""
-echo "Darwin (macOS) amd64"
-GOOS=darwin GOARCH=amd64 go build -o dist/darwin-amd64/watchdog $gocmd $golib
-
-echo ""
-echo "Linux amd64"
-GOOS=linux GOARCH=amd64 go build -o dist/linux-amd64/watchdog $gocmd $golib
-echo "Linux 386"
-
-echo ""
-GOOS=linux GOARCH=386 go build -o dist/linux-386/watchdog $gocmd $golib
-echo "RPi 3 B+ ARMv7"
-GOOS=linux GOARCH=arm GOARM=7 go build -o dist/linux-armv7/watchdog $gocmd $golib
-echo "RPi Zero ARMv5"
-GOOS=linux GOARCH=arm GOARM=5 go build -o dist/linux-armv5/watchdog $gocmd $golib
-
-my_ver=$(git describe --tags)
-pushd dist
-  ls -d *-* | while read my_dist
-  do
-    if [ -d "$my_dist" ]; then
-      #tar -czvf watchdog-$my_ver-$my_dist.tar.gz $my_dist
-      zip -r watchdog-$my_ver-$my_dist.zip $my_dist
-    fi
-  done
-popd
-
-echo ""
-echo ""
--- a/cmd/watchdog/watchdog.go
+++ b/cmd/watchdog/watchdog.go
@ -11,7 +11,7 @@ import (
 	"os"
 	"strings"

-	watchdog "git.rootprojects.org/root/watchdog.go"
+	watchdog "git.rootprojects.org/root/go-watchdog"
 )

 var GitRev, GitVersion, GitTimestamp string
@ -83,9 +83,12 @@ func main() {
 		logQueue <- fmt.Sprintf("Watching '%s'", c.Name)
 		go func(c watchdog.ConfigWatch) {
 			d := watchdog.New(&watchdog.Dog{
+				Watchdog:      config.Watchdog,
 				Name:          c.Name,
 				CheckURL:      c.URL,
 				Keywords:      c.Keywords,
+				Badwords:      c.Badwords,
+				Localizations: config.Localizations,
 				Recover:       c.RecoverScript,
 				Webhooks:      c.Webhooks,
 				AllWebhooks:   allWebhooks,
--- a/doc.go
+++ b/doc.go
@ -5,5 +5,5 @@
 // The git tag version describes the state of the binary,
 // not the state of the library. The API is not yet stable.
 //
-// See https://git.rootproject.org/root/watchdog.go for pre-built binaries.
+// See https://git.rootproject.org/root/go-watchdog for pre-built binaries.
 package watchdog
--- a/go.mod
+++ b/go.mod
@ -1,4 +1,4 @@
-module git.rootprojects.org/root/watchdog.go
+module git.rootprojects.org/root/go-watchdog

 go 1.12

--- a/watchdog.go
+++ b/watchdog.go
@ -14,14 +14,38 @@ import (
 	"time"
 )

+type Status int
+
+const (
+	StatusDown Status = iota
+	StatusUp
+)
+
+func (s Status) String() string {
+	// ... just wishing Go had enums like Rust...
+	switch s {
+	case StatusUp:
+		return "up"
+	case StatusDown:
+		return "down"
+	default:
+		return "[[internal error]]"
+	}
+}
+
 type Dog struct {
+	Watchdog      string
 	Name          string
 	CheckURL      string
 	Keywords      string
+	Badwords      string
+	Localizations map[string]string
 	Recover       string
 	Webhooks      []string
 	AllWebhooks   map[string]Webhook
 	Logger        chan string
+	status        Status
+	changed       bool
 	error         error
 	failures      int
 	passes        int
@ -32,6 +56,8 @@ type Dog struct {

 func New(d *Dog) *Dog {
 	d.lastPassed = time.Now().Add(-5 * time.Minute)
+	d.status = StatusUp
+	d.changed = false
 	return d
 }

@ -49,10 +75,14 @@ func (d *Dog) watch() {

 	err := d.check()
 	if nil == err {
+		if d.changed {
+			d.notify("came back up")
+		}
 		return
 	}

-	time.Sleep(time.Duration(2) * time.Second)
+	time.Sleep(time.Duration(5) * time.Second)
+
 	err2 := d.check()
 	if nil != err2 {
 		d.Logger <- fmt.Sprintf("Down: '%s': %s", d.Name, err2)
@ -61,7 +91,6 @@ func (d *Dog) watch() {
 		return
 	}

-	failure := false
 	t := 10
 	for {
 		d.recover()
@ -71,34 +100,55 @@ func (d *Dog) watch() {
 		err := d.check()
 		if nil != err {
 			d.Logger <- fmt.Sprintf("Unrecoverable: '%s': %s", d.Name, err)
-			failure = true
-		} else {
-			failure = false
 		}

 		// We should notify if
+		// * The status has changed
+		//
+		// TODO what if the server is flip-flopping rapidly?
+		// how to rate limit?
+		// "{{ .Server }} is on cooldown for 30 minutes"
+		if d.changed {
+			d.notify("went down")
+			if StatusUp == d.status {
+				break
+			}
+
 			// * We've had success since the last notification
 			// * It's been at least 5 minutes since the last notification
-		fiveMinutesAgo := time.Now().Add(-5 * time.Minute)
-		if d.lastPassed.After(d.lastNotified) && d.lastNotified.Before(fiveMinutesAgo) {
-			d.notify(failure)
-		}
-		if !failure || d.failures >= 5 {
+			//fiveMinutesAgo := time.Now().Add(-5 * time.Minute)
+			//if d.lastPassed.After(d.lastNotified) && d.lastNotified.Before(fiveMinutesAgo) {
+			//}
+			//if !failure || d.failures >= 5 {
 			// go back to the main 5-minute loop
-			break
+			//	break
+			//}
 		}
 	}
 }

 func (d *Dog) check() error {
+	previousStatus := d.status
+
 	var err error
 	defer func() {
+		// Are we up, or down?
 		if nil != err {
+			d.status = StatusDown
 			d.failures += 1
 			d.lastFailed = time.Now()
 		} else {
+			d.status = StatusUp
 			d.lastPassed = time.Now()
 			d.passes += 1
+			d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
+		}
+
+		// Has that changed?
+		if previousStatus != d.status {
+			d.changed = true
+		} else {
+			d.changed = false
 		}
 	}()

@ -115,13 +165,21 @@ func (d *Dog) check() error {
 		return err
 	}

+	// Note: empty matches empty as true, so this works for checking redirects
 	if !bytes.Contains(b, []byte(d.Keywords)) {
 		err = fmt.Errorf("Down: '%s' Not Found for '%s'", d.Keywords, d.Name)
 		d.Logger <- fmt.Sprintf("%s", err)
 		d.error = err
 		return err
-	} else {
-		d.Logger <- fmt.Sprintf("Up: '%s'", d.Name)
+	}
+
+	if "" != d.Badwords {
+		if !bytes.Contains(b, []byte(d.Badwords)) {
+			err = fmt.Errorf("Down: '%s' Found for '%s'", d.Badwords, d.Name)
+			d.Logger <- fmt.Sprintf("%s", err)
+			d.error = err
+			return err
+		}
 	}

 	return nil
@ -154,8 +212,8 @@ func (d *Dog) recover() {
 	}
 }

-func (d *Dog) notify(hardFail bool) {
-	d.Logger <- fmt.Sprintf("Notifying the authorities of %s's failure", d.Name)
+func (d *Dog) notify(msg string) {
+	d.Logger <- fmt.Sprintf("Notifying the authorities of %s's status change", d.Name)
 	d.lastNotified = time.Now()

 	for i := range d.Webhooks {
@ -172,11 +230,11 @@ func (d *Dog) notify(hardFail bool) {
 			continue
 		}

-		d.notifyOne(h, hardFail)
+		d.notifyOne(h, msg)
 	}
 }

-func (d *Dog) notifyOne(h Webhook, hardFail bool) {
+func (d *Dog) notifyOne(h Webhook, msg string) {
 	// TODO do this in main on config init
 	if "" == h.Method {
 		h.Method = "POST"
@ -191,7 +249,10 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
 			v := h.Form[k]
 			// because `{{` gets urlencoded
 			//k = strings.Replace(k, "{{ .Name }}", d.Name, -1)
+			v = strings.Replace(v, "{{ .Watchdog }}", d.Watchdog, -1)
 			v = strings.Replace(v, "{{ .Name }}", d.Name, -1)
+			v = strings.Replace(v, "{{ .Status }}", d.localize(d.status.String()), -1)
+			v = strings.Replace(v, "{{ .Message }}", d.localize(msg), -1)
 			d.Logger <- fmt.Sprintf("[HEADER] %s: %s", k, v)
 			form.Set(k, v)
 		}
@ -203,7 +264,12 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
 			return
 		}
 		// `{{` should be left alone
-		body = strings.NewReader(strings.Replace(string(bodyBuf), "{{ .Name }}", d.Name, -1))
+		v := string(bodyBuf)
+		v = strings.Replace(v, "{{ .Watchdog }}", d.Watchdog, -1)
+		v = strings.Replace(v, "{{ .Name }}", d.Name, -1)
+		v = strings.Replace(v, "{{ .Status }}", d.localize(d.status.String()), -1)
+		v = strings.Replace(v, "{{ .Message }}", d.localize(msg), -1)
+		body = strings.NewReader(v)
 	}

 	client := NewHTTPClient()
@ -260,16 +326,27 @@ func (d *Dog) notifyOne(h Webhook, hardFail bool) {
 	// TODO some sort of way to determine if data is successful (keywords)
 	d.Logger <- fmt.Sprintf("[Notify] Success? %#v", data)
 }
+func (d *Dog) localize(msg string) string {
+	for k := range d.Localizations {
+		if k == msg {
+			return d.Localizations[k]
+		}
+	}
+	return msg
+}

 type Config struct {
+	Watchdog      string            `json:"watchdog"`
 	Watches       []ConfigWatch     `json:"watches"`
 	Webhooks      []Webhook         `json:"webhooks"`
+	Localizations map[string]string `json:"localizations"`
 }

 type ConfigWatch struct {
 	Name          string   `json:"name"`
 	URL           string   `json:"url"`
 	Keywords      string   `json:"keywords"`
+	Badwords      string   `json:"badwords"`
 	Webhooks      []string `json:"webhooks"`
 	RecoverScript string   `json:"recover_script"`
 }