goldilocks.js/README.md

Goldilocks
==========

The node.js netserver that's just right.

* **HTTPS Web Server** with Automatic TLS (SSL) via ACME ([Let's Encrypt](https://letsencrypt.org))
  * Static Web Server
  * URL Redirects
  * SSL on localhost (with bundled localhost.daplie.me certificates)
  * Uses node cluster to take advantage of multiple CPUs (in progress)
* **TLS** name-based (SNI) proxy
* **TCP** port-based proxy
* WS **Tunnel Server** (i.e. run on Digital Ocean and expose a home-firewalled Raspberry Pi to the Internet)
* WS **Tunnel Client** (i.e. run on a Raspberry Pi and connect to a Daplie Tunnel)
* UPnP / NAT-PMP forwarding and loopback testing (in progress)
* Configurable via API
* mDNS Discoverable (configure in home or office with mobile and desktop apps)
* OAuth3 Authentication

Install Standalone
-------

### curl | bash

```bash
curl -fsSL https://git.coolaj86.com/coolaj86/goldilocks.js/raw/v1.1/installer/get.sh | bash
```

### git

```bash
git clone https://git.coolaj86.com/coolaj86/goldilocks.js
pushd goldilocks.js
git checkout v1.1
bash installer/install.sh
```

### npm

```bash
# v1 in git (unauthenticated)
npm install -g git+https://git@git.coolaj86.com:coolaj86/goldilocks.js#v1

# v1 in git (via ssh)
npm install -g git+ssh://git@git.coolaj86.com:coolaj86/goldilocks.js#v1

# v1 in npm
npm install -g goldilocks@v1
```

### Uninstall

Remove goldilocks and services:

```
rm -rf /opt/goldilocks/ /srv/goldilocks/ /var/goldilocks/ /var/log/goldilocks/ /etc/tmpfiles.d/goldilocks.conf /etc/systemd/system/goldilocks.service
```

Remove config as well

```
rm -rf /etc/goldilocks/ /etc/ssl/goldilocks
```

Usage
-----

```bash
goldilocks
```

```bash
Serving /Users/foo/ at https://localhost.daplie.me:8443
```

Install as a System Service (daemon-mode)

We have service support for

* systemd (Linux, Ubuntu)
* launchd (macOS)

```bash
curl https://git.coolaj86.com/coolaj86/goldilocks.js/raw/master/install.sh | bash
```

Modules & Configuration
-----

Goldilocks has several core systems, which all have their own configuration and
some of which have modules:

* [http](#http)
  - [proxy (reverse proxy)](#httpproxy-how-to-reverse-proxy-ruby-python-etc)
  - [static](#httpstatic-how-to-serve-a-web-page)
  - [redirect](#httpredirect-how-to-redirect-urls)
* [tls](#tls)
  - [proxy (reverse proxy)](#tlsproxy)
  - [acme](#tlsacme)
* [tcp](#tcp)
  - [proxy](#tcpproxy)
  - [forward](#tcpforward)
* [udp](#udp)
  - [forward](#udpforward)
* [domains](#domains)
* [tunnel_server](#tunnel_server)
* [DDNS](#ddns)
* [tunnel_client](#tunnel)
* [mDNS](#mdns)
* [socks5](#socks5)
* api

All modules require a `type` and an `id`, and any modules not defined inside the
`domains` system also require a `domains` field (with the exception of the `forward`
modules that require the `ports` field).

### http

The HTTP system handles plain http (TLS / SSL is handled by the tls system)

Example config:
```yml
http:
  trust_proxy: true                 # allow localhost, 192.x, 10.x, 172.x, etc to set headers
  allow_insecure: false             # allow non-https even without proxy https headers
  primary_domain: example.com       # attempts to access via IP address will redirect here

  # An array of modules that define how to handle incoming HTTP requests
  modules:
    - type: static
      domains:
        - example.com
      root: /srv/www/:hostname
```

### http.proxy - how to reverse proxy (ruby, python, etc)

The proxy module is for reverse proxying, typically to an application on the same machine.
(Though it can also reverse proxy to other devices on the local network.)

It has the following options:
```
address     The DNS-resolvable hostname (or IP address) and port connected by `:` to proxy the request to.
            Takes priority over host and port if they are also specified.
            ex: locahost:3000
            ex: 192.168.1.100:80

host        The DNS-resolvable hostname (or IP address) of the system to which the request will be proxied.
            Defaults to localhost if only the port is specified.
            ex: localhost
            ex: 192.168.1.100

port        The port on said system to which the request will be proxied
            ex: 3000
            ex: 80
```

Example config:
```yml
http:
  modules:
    - type: proxy
      domains:
        - api.example.com
      host: 192.168.1.100
      port: 80
    - type: proxy
      domains:
        - www.example.com
      address: 192.168.1.16:80
    - type: proxy
      domains:
        - '*'
      port: 3000
```

### http.static - how to serve a web page

The static module is for serving static web pages and assets and has the following options:

```
root        The path to serve as a string.
            The template variable `:hostname` represents the HTTP Host header without port information
            ex: `root: /srv/www/example.com` would load the example.com folder for any domain listed
            ex: `root: /srv/www/:hostname` would load `/srv/www/example.com` if so indicated by the Host header

index       Set to `false` to disable the default behavior of loading `index.html` in directories
            ex: `false`

dotfiles    Set to `allow` to load dotfiles rather than ignoring them
            ex: `"allow"`

redirect    Set to `false` to disable the default behavior of ensuring that directory paths end in '/'
            ex: `false`

indexes     An array of directories which should be have indexes served rather than blocked
            ex: `[ '/' ]` will allow all directories indexes to be served
```

Example config:
```yml
http:
  modules:
    - type: static
      domains:
        - example.com
      root: /srv/www/:hostname
```

### http.redirect - how to redirect URLs

The redirect module is for, you guessed it, redirecting URLs.

It has the following options:
```
status      The HTTP status code to issue (301 is usual permanent redirect, 302 is temporary)
            ex: 301

from        The URL path that was used in the request.
            The `*` wildcard character can be used for matching a full segment of the path
            ex: /photos/
            ex: /photos/*/*/

to          The new URL path which should be used.
            If wildcards matches were used they will be available as `:1`, `:2`, etc.
            ex: /pics/
            ex: /pics/:1/:2/
            ex: https://mydomain.com/photos/:1/:2/
```

Example config:
```yml
http:
  modules:
    - type: proxy
      domains:
        - example.com
      status: 301
      from: /archives/*/*/*/
      to: https://example.net/year/:1/month/:2/day/:3/
```

### tls

The tls system handles encrypted connections, including fetching certificates,
and uses ServerName Indication (SNI) to determine if the connection should be
handled by the http system, a tls system module, or rejected.

Example config:
```yml
tls:
  modules:
    - type: proxy
      domains:
        - example.com
        - example.net
      address: '127.0.0.1:6443'
```

Certificates are saved to `~/acme`, which may be `/var/www/acme` if Goldilocks is run as the www-data user.

### tls.proxy

The proxy module routes the traffic based on the ServerName Indication (SNI) **without decrypting** it.

It has the same options as the [HTTP proxy module](#httpproxy-how-to-reverse-proxy-ruby-python-etc).

Example config:
```yml
tls:
  modules:
    - type: proxy
      domains:
        - example.com
      address: '127.0.0.1:5443'
```

### tls.acme

The acme module defines the setting used when getting new certificates.

It has the following options:
```
email              The email address for ACME certificate issuance
                   ex: john.doe@example.com

server             The ACME server to use
                   ex: https://acme-v01.api.letsencrypt.org/directory
                   ex: https://acme-staging.api.letsencrypt.org/directory

challenge_type     The ACME challenge to request
                   ex: http-01, dns-01, tls-01
```

Example config:
```yml
tls:
  modules:
    - type: acme
      domains:
        - example.com
        - example.net
      email: 'joe.shmoe@example.com'
      server: 'https://acme-staging.api.letsencrypt.org/directory'
      challenge_type: 'http-01'
```

**NOTE:** If you specify `dns-01` as the challenge type there must also be a
[DDNS module](#ddns) defined for all of the relevant domains (though not all
domains handled by a single TLS module need to be handled by the same DDNS
module). The DDNS module provides all of the information needed to actually
set the DNS records needed to verify ownership.

### tcp

The tcp system handles both *raw* and *tls-terminated* tcp network traffic
(see the _Note_ section below the example). It may use port numbers
or traffic sniffing to determine how the connection should be handled.

It has the following options:
```
bind      An array of numeric ports on which to bind
          ex: 80
```

Example Config:
```yml
tcp:
  bind:
    - 22
    - 80
    - 443
  modules:
    - type: forward
      ports:
        - 22
      address: '127.0.0.1:2222'
```

_Note_: When tcp traffic comes into goldilocks it will be tested against the tcp modules.
The connection may be handed to the TLS module if it appears to be a TLS/SSL/HTTPS connection
and if the tls module terminates the traffic, the connection will be sent back to the TLS module.
Due to the complexity of node.js' networking stack it is not currently possible to tell which
port tls-terminated traffic came from, so only the SNI header (serername / domain name) may be used for
modules matching terminated TLS.

### tcp.proxy

The proxy module routes traffic **after tls-termination** based on the servername (domain name)
contained in a SNI header. As such this only works to route TCP connections wrapped in a TLS stream.

It has the same options as the [HTTP proxy module](#httpproxy-how-to-reverse-proxy-ruby-python-etc).

This is particularly useful for routing ssh and vpn traffic over tcp port 443 as wrapped TLS
connections in order to access one of your servers even when connecting from a harsh or potentially
misconfigured network environment (i.e. hotspots in public libraries and shopping malls).

Example config:
```yml
tcp:
  modules:
    - type: proxy
      domains:
        - ssh.example.com      # Note: this domain would also listed in tls.acme.domains
      host: localhost
      port: 22
    - type: proxy
      domains:
        - vpn.example.com      # Note: this domain would also listed in tls.acme.domains
      host: localhost
      port: 1194
```

_Note_: In same cases network administrators purposefully block ssh and vpn connections using
Application Firewalls with DPI (deep packet inspection) enabled. You should read the ToS of the
network you are connected to to ensure that you aren't subverting policies that are purposefully
in place on such networks.

#### Using with ssh

In order to use this to route SSH connections you will need to use `ssh`'s
`ProxyCommand` option. For example to use the TLS certificate for `ssh.example.com`
to wrap an ssh connection you could use the following command:

```bash
ssh user@example.com -o ProxyCommand='openssl s_client -quiet -connect example.com:443 -servername ssh.example.com'
```

Alternatively you could add the following lines to your ssh config file.
```
Host example.com
  ProxyCommand openssl s_client -quiet -connect example.com:443 -servername ssh.example.com
```

#### Using with OpenVPN

There are two strategies that will work well for you:

1) [Use ssh](https://redfern.me/tunneling-openvpn-through-ssh/) with the config above to reverse proxy tcp port 1194 to you.

```bash
ssh -L 1194:localhost:1194 example.com
```

2) [Use stunnel]https://serverfault.com/questions/675553/stunnel-vpn-traffic-and-ensure-it-looks-like-ssl-traffic-on-port-443/681497)

```
[openvpn-over-goldilocks]
client = yes
accept = 127.0.0.1:1194
sni = vpn.example.com
connect = example.com:443
```

3) [Use stunnel.js](https://git.coolaj86.com/coolaj86/tunnel-client.js) as described in the "tunnel_server" section below.

### tcp.forward

The forward module routes traffic based on port number **without decrypting** it.

In addition to the same options as the [HTTP proxy module](#httpproxy-how-to-reverse-proxy-ruby-python-etc),
the TCP forward modules also has the following options:

```
ports       A numeric array of source ports
            ex: 22
```

Example Config:
```yml
tcp:
  bind:
    - 22
    - 80
    - 443
  modules:
    - type: forward
      ports:
        - 22
      port: 2222
```

### udp

The udp system handles all udp network traffic. It currently only supports
forwarding the messages without any examination.

It has the following options:
```
bind      An array of numeric ports on which to bind
          ex: 53
```

Example Config:
```yml
udp:
  bind:
    - 53
  modules:
    - type: forward
      ports:
        - 53
      address: '127.0.0.1:8053'
```

### udp.forward

The forward module routes traffic based on port number **without decrypting** it.

It has the same options as the [TCP forward module](#tcpforward).

Example Config:
```yml
udp:
  bind:
    - 53
  modules:
    - type: forward
      ports:
        - 53
      address: '127.0.0.1:8053'
```

### domains

To reduce repetition defining multiple modules that operate on the same domain
name the `domains` field can define multiple modules of multiple types for a
single list of names. The modules defined this way do not need to have their
own `domains` field. Note that the [tcp.forward](#tcpforward) module is not
allowed in a domains group since its routing is not based on domains.

Example Config

```yml
domains:
  - names:
      - example.com
      - www.example.com
      - api.example.com
    modules:
      tls:
        - type: acme
          email: joe.schmoe@example.com
          challenge_type: 'http-01'
      http:
        - type: redirect
          from: /deprecated/path
          to: /new/path
        - type: proxy
          port: 3000
      dns:
        - type: 'dns@oauth3.org'
          token_id: user_token_id

  - names:
      - ssh.example.com
    modules:
      tls:
        - type: acme
          email: john.smith@example.com
          challenge_type: 'http-01'
      tcp:
        - type: proxy
          port: 22
      dns:
        - type: 'dns@oauth3.org'
          token_id: user_token_id
```



### tunnel\_server

The tunnel server system is meant to be run on a publicly accessible IP address to server tunnel clients
which are behind firewalls, carrier-grade NAT, or otherwise Internet-connect but inaccessible devices.

It has the following options:

```
secret          A 128-bit or greater string to use for signing tokens (HMAC JWT)
                ex: abc123

servernames     An array of string servernames that should be captured as the
                tunnel server, ignoring the TLS forward module
                ex: api.tunnel.example.com
```

Example config:

```yml
tunnel_server:
  secret: abc123def456ghi789
  servernames:
    - 'api.tunnel.example.com'
```

### DDNS

The DDNS module watches the network environment of the unit and makes sure the
device is always accessible on the internet using the domains listed in the
config. If the device has a public address or if it can automatically set up
port forwarding the device will periodically check its public address to ensure
the DNS records always point to it. Otherwise it will to connect to a tunnel
server and set the DNS records to point to that server.

The `loopback` setting specifies how the unit will check its public IP address
and whether connections can reach it. Currently only `tunnel@oauth3.org` is
supported. If the loopback setting is not defined it will default to using
`oauth3.org`.

The `tunnel` setting can be used to specify how to connect to the tunnel.
Currently only `tunnel@oauth3.org` is supported. The token specified in the
`tunnel` setting will be used to acquire the tokens that are used directly with
the tunnel server. If the tunnel setting is not defined it will default to try
using the tokens in the modules for the relevant domains.

If a particular DDNS module has been disabled the device will still try to set
up port forwarding (and connect to a tunnel if that doesn't work), but the DNS
records will not be updated to point to the device. This is to allow a setup to
be tested before transitioning services between devices.

```yaml
ddns:
  disabled: false
  loopback:
    type: 'tunnel@oauth3.org'
    domain: oauth3.org
  tunnel:
    type: 'tunnel@oauth3.org'
    token_id: user_token_id
  modules:
    - type: 'dns@oauth3.org'
      token_id: user_token_id
      domains:
        - www.example.com
        - api.example.com
        - test.example.com
```

### mDNS

enabled by default

Although it does not announce itself, Goldilocks is discoverable via mDNS with the special query `_cloud._tcp.local`.
This is so that it can be easily configured via Desktop and Mobile apps when run on devices such as a Raspberry Pi or
SOHO servers.

```yaml
mdns:
  disabled: false
  port: 5353
  broadcast: '224.0.0.251'
  ttl: 300
```

You can discover goldilocks with `mdig`.

```
npm install -g git+https://git.coolaj86.com/coolaj86/mdig.js.git

mdig _cloud._tcp.local
```

### socks5

Run a Socks5 proxy server.

```yaml
socks5:
  enable: true
  port: 1080
```

### api

See [API.md](/API.md)

@tigerbot: How are the APIs used (in terms of URL, Method, Headers, etc)?

TODO
----

* [ ] http - nowww module
* [ ] http - Allow match styles of `www.*`, `*`, and `*.example.com` equally
* [ ] http - redirect based on domain name (not just path)
* [ ] tcp - bind should be able to specify localhost, uniquelocal, private, or ip
* [ ] tcp - if destination host is omitted default to localhost, if dst port is missing, default to src
* [ ] sys - `curl https://coolaj86.com/goldilocks | bash -s example.com`
* [ ] oauth3 - `example.com/.well-known/domains@oauth3.org/directives.json`
* [ ] oauth3 - commandline questionnaire
* [x] modules - use consistent conventions (i.e. address vs host + port)
  * [x] tls - tls.acme vs tls.modules.acme
* [ ] tls - forward should be able to match on source port to reach different destination ports