diff --git a/API.md b/API.md index 6495f06..fe075fa 100644 --- a/API.md +++ b/API.md @@ -2,13 +2,113 @@ The API system is intended for use with Desktop and Mobile clients. It must be accessed using one of the following domains as the Host header: -``` -admin.invalid -localhost.admin.daplie.me -``` +* localhost.alpha.daplie.me +* localhost.admin.daplie.me +* alpha.localhost.daplie.me +* admin.localhost.daplie.me +* localhost.daplie.invalid All requests require an OAuth3 token in the request headers. +## Config + +### Get All Settings + * **URL** `/api/goldilocks@daplie.com/config` + * **Method** `GET` + * **Reponse**: The JSON representation of the current config. See the [README.md](/README.md) + for the structure of the config. + +### Get Group Setting + * **URL** `/api/goldilocks@daplie.com/config/:group` + * **Method** `GET` + * **Reponse**: The sub-object of the config relevant to the group specified in + the url (ie http, tls, tcp, etc.) + +### Get Group Module List + * **URL** `/api/goldilocks@daplie.com/config/:group/modules` + * **Method** `GET` + * **Reponse**: The list of modules relevant to the group specified in the url + (ie http, tls, tcp, etc.) + +### Get Specific Module + * **URL** `/api/goldilocks@daplie.com/config/:group/modules/:modId` + * **Method** `GET` + * **Reponse**: The module with the specified module ID. + +### Get Domain Group + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId` + * **Method** `GET` + * **Reponse**: The domains specification with the specified domains ID. + +### Get Domain Group Modules + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId/modules` + * **Method** `GET` + * **Reponse**: An object containing all of the relevant modules for the group + of domains. + +### Get Domain Group Module Category + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId/modules/:group` + * **Method** `GET` + * **Reponse**: A list of the specific category of modules for the group of domains. + +### Get Specific Domain Group Module + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId/modules/:group/:modId` + * **Method** `GET` + * **Reponse**: The module with the specified module ID. + + +### Change Settings + * **URL** `/api/goldilocks@daplie.com/config` + * **URL** `/api/goldilocks@daplie.com/config/:group` + * **Method** `POST` + * **Body**: The changes to be applied on top of the current config. See the + [README.md](/README.md) for the settings. If modules or domains are specified + they are added to the current list. + * **Reponse**: The current config. If the group is specified in the URL it will + only be the config relevant to that group. + +### Add Module + * **URL** `/api/goldilocks@daplie.com/config/:group/modules` + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId/modules/:group` + * **Method** `POST` + * **Body**: The module to be added. Can also be provided an array of modules + to add multiple modules in the same request. + * **Reponse**: The current list of modules. + +### Add Domain Group + * **URL** `/api/goldilocks@daplie.com/config/domains` + * **Method** `POST` + * **Body**: The domains names and modules for the new domain group(s). + * **Reponse**: The current list of domain groups. + + +### Edit Module + * **URL** `/api/goldilocks@daplie.com/config/:group/modules/:modId` + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId/modules/:group/:modId` + * **Method** `PUT` + * **Body**: The new parameters for the module. + * **Reponse**: The editted module. + +### Edit Domain Group + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId` + * **Method** `PUT` + * **Body**: The new domains names for the domains group. The module list cannot + be editted through this route. + * **Reponse**: The editted domain group. + + +### Remove Module + * **URL** `/api/goldilocks@daplie.com/config/:group/modules/:modId` + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId/modules/:group/:modId` + * **Method** `DELETE` + * **Reponse**: The list of modules. + +### Remove Domain Group + * **URL** `/api/goldilocks@daplie.com/config/domains/:domId` + * **Method** `DELETE` + * **Reponse**: The list of domain groups. + + ## Socks5 Proxy ### Check Status @@ -21,9 +121,9 @@ All requests require an OAuth3 token in the request headers. ### Start Proxy * **URL** `/api/goldilocks@daplie.com/socks5` * **Method** `POST` - * **Response**: Same response as for the `GET` resquest + * **Response**: Same response as for the `GET` request ### Stop Proxy * **URL** `/api/goldilocks@daplie.com/socks5` * **Method** `DELETE` - * **Response**: Same response as for the `GET` resquest + * **Response**: Same response as for the `GET` request diff --git a/README.md b/README.md index 4491569..54b7931 100644 --- a/README.md +++ b/README.md @@ -53,50 +53,91 @@ curl https://git.daplie.com/Daplie/goldilocks.js/raw/master/install.sh | bash Modules & Configuration ----- -Goldilocks has several core systems, which all have their own configuration and some of which have modules: +Goldilocks has several core systems, which all have their own configuration and +some of which have modules: -``` -* http - - static - - redirect - - proxy (reverse proxy) -* tls - - acme - - proxy (reverse proxy) -* tcp - - forward -* tunnel_server -* tunnel_client -* mdns +* [http](#http) + - [proxy (reverse proxy)](#httpproxy-how-to-reverse-proxy-ruby-python-etc) + - [static](#httpstatic-how-to-serve-a-web-page) + - [redirect](#httpredirect-how-to-redirect-urls) +* [tls](#tls) + - [proxy (reverse proxy)](#tlsproxy) + - [acme](#tlsacme) +* [tcp](#tcp) + - [forward](#tcpforward) +* [udp](#udp) + - [forward](#udpforward) +* [domains](#domains) +* [tunnel_server](#tunnel_server) +* [tunnel_client](#tunnel) +* [mdns](#mdns) +* [socks5](#socks5) * api -``` + +All modules require a `type` and an `id`, and any modules not defined inside the +`domains` system also require a `domains` field (with the exception of the `forward` +modules that require the `ports` field). ### http The HTTP system handles plain http (TLS / SSL is handled by the tls system) +Example config: ```yml http: trust_proxy: true # allow localhost, 192.x, 10.x, 172.x, etc to set headers allow_insecure: false # allow non-https even without proxy https headers primary_domain: example.com # attempts to access via IP address will redirect here - # modules can be nested in domains - domains: - - names: - - example.com - modules: - - name: static - root: /srv/www/:hostname - - # The configuration above could also be represented as follows: + # An array of modules that define how to handle incoming HTTP requests modules: - - name: static + - type: static domains: - example.com root: /srv/www/:hostname ``` +### http.proxy - how to reverse proxy (ruby, python, etc) + +The proxy module is for reverse proxying, typically to an application on the same machine. +(Though it can also reverse proxy to other devices on the local network.) + +It has the following options: +``` +address The DNS-resolvable hostname (or IP address) and port connected by `:` to proxy the request to. + Takes priority over host and port if they are also specified. + ex: locahost:3000 + ex: 192.168.1.100:80 + +host The DNS-resolvable hostname (or IP address) of the system to which the request will be proxied. + Defaults to localhost if only the port is specified. + ex: localhost + ex: 192.168.1.100 + +port The port on said system to which the request will be proxied + ex: 3000 + ex: 80 +``` + +Example config: +```yml +http: + modules: + - type: proxy + domains: + - api.example.com + host: 192.168.1.100 + port: 80 + - type: proxy + domains: + - www.example.com + address: 192.168.1.16:80 + - type: proxy + domains: + - '*' + port: 3000 +``` + ### http.static - how to serve a web page The static module is for serving static web pages and assets and has the following options: @@ -109,50 +150,20 @@ root The path to serve as a string. ``` Example config: - ```yml http: modules: - - name: static + - type: static domains: - example.com root: /srv/www/:hostname ``` -### http.proxy - how to reverse proxy (ruby, python, etc) - -The proxy module is for reverse proxying, typically to an application on the same machine. - -It has the following options: - -``` -host The DNS-resolvable hostname (or IP address) of the system to which the request will be proxied - ex: localhost - ex: 192.168.1.100 - -port The port on said system to which the request will be proxied - ex: 3000 - ex: 80 -``` - -Example config: - -```yml -http: - modules: - - name: proxy - domains: - - example.com - host: localhost - port: 3000 -``` - ### http.redirect - how to redirect URLs The redirect module is for, you guessed it, redirecting URLs. It has the following options: - ``` status The HTTP status code to issue (301 is usual permanent redirect, 302 is temporary) ex: 301 @@ -169,11 +180,10 @@ to The new URL path which should be used. ``` Example config: - ```yml http: modules: - - name: proxy + - type: proxy domains: - example.com status: 301 @@ -184,41 +194,14 @@ http: ### tls The tls system handles encrypted connections, including fetching certificates, -and uses ServerName Indication (SNI) to determine if the connection should be handled -by the http system, a tls system module, or rejected. - -It has the following options: - -``` -acme.email The default email address for ACME certificate issuance - ex: john.doe@example.com - -acme.server The default ACME server to use - ex: https://acme-v01.api.letsencrypt.org/directory - ex: https://acme-staging.api.letsencrypt.org/directory - -acme.challenge_type The default ACME challenge to request - ex: http-01, dns-01, tls-01 - -acme.approved_domains The domains for which to request certificates - ex: example.com -``` +and uses ServerName Indication (SNI) to determine if the connection should be +handled by the http system, a tls system module, or rejected. Example config: - ```yml tls: - acme: - email: 'joe.shmoe@example.com' - # IMPORTANT: Switch to in production 'https://acme-v01.api.letsencrypt.org/directory' - server: 'https://acme-staging.api.letsencrypt.org/directory' - challenge_type: 'http-01' - approved_domains: - - example.com - - example.net - modules: - - name: proxy + - type: proxy domains: - example.com - example.net @@ -227,17 +210,44 @@ tls: Certificates are saved to `~/acme`, which may be `/var/www/acme` if Goldilocks is run as the www-data user. -### tls.acme +### tls.proxy -The acme module overrides the acme defaults of the tls system and uses the same options except that `approved_domains` -(in favor of the domains in the scope of the module). +The proxy module routes the traffic based on the ServerName Indication (SNI) **without decrypting** it. + +It has the same options as the [HTTP proxy module](#httpproxy-how-to-reverse-proxy-ruby-python-etc). Example config: - ```yml tls: modules: - - name: acme + - type: proxy + domains: + - example.com + address: '127.0.0.1:5443' +``` + +### tls.acme + +The acme module defines the setting used when getting new certificates. + +It has the following options: +``` +email The email address for ACME certificate issuance + ex: john.doe@example.com + +server The ACME server to use + ex: https://acme-v01.api.letsencrypt.org/directory + ex: https://acme-staging.api.letsencrypt.org/directory + +challenge_type The ACME challenge to request + ex: http-01, dns-01, tls-01 +``` + +Example config: +```yml +tls: + modules: + - type: acme domains: - example.com - example.net @@ -246,41 +256,18 @@ tls: challenge_type: 'http-01' ``` -### tls.proxy - -The proxy module routes the traffic based on the ServerName Indication (SNI) **without decrypting** it. - -It has the following options: - -``` -address The hostname (or IP) and port of the system or application that should receive the traffic -``` - -Example config: - -```yml -tls: - modules: - - name: proxy - domains: - - example.com - address: '127.0.0.1:5443' -``` - ### tcp The tcp system handles all tcp network traffic **before decryption** and may use port numbers or traffic sniffing to determine how the connection should be handled. It has the following options: - ``` bind An array of numeric ports on which to bind ex: 80 ``` -Example Config - +Example Config: ```yml tcp: bind: @@ -288,7 +275,7 @@ tcp: - 80 - 443 modules: - - name: forward + - type: forward ports: - 22 address: '127.0.0.1:2222' @@ -298,18 +285,15 @@ tcp: The forward module routes traffic based on port number **without decrypting** it. -It has the following options: +In addition to the same options as the [HTTP proxy module](#httpproxy-how-to-reverse-proxy-ruby-python-etc), +the TCP forward modules also has the following options: ``` ports A numeric array of source ports ex: 22 - -address The destination hostname and port - ex: 127.0.0.1:2222 ``` -Example Config - +Example Config: ```yml tcp: bind: @@ -317,10 +301,79 @@ tcp: - 80 - 443 modules: - - name: forward + - type: forward ports: - 22 - address: '127.0.0.1:2222' + port: 2222 +``` + +### udp + +The udp system handles all udp network traffic. It currently only supports +forwarding the messages without any examination. + +It has the following options: +``` +bind An array of numeric ports on which to bind + ex: 53 +``` + +Example Config: +```yml +udp: + bind: + - 53 + modules: + - type: forward + ports: + - 53 + address: '127.0.0.1:8053' +``` + +### udp.forward + +The forward module routes traffic based on port number **without decrypting** it. + +It has the same options as the [TCP forward module](#tcpforward). + +Example Config: +```yml +udp: + bind: + - 53 + modules: + - type: forward + ports: + - 53 + address: '127.0.0.1:8053' +``` + +### domains + +To reduce repetition defining multiple modules that operate on the same domain +name the `domains` field can define multiple modules of multiple types for a +single list of names. The modules defined this way do not need to have their +own `domains` field. + +Example Config + +```yml +domains: + names: + - example.com + - www.example.com + - api.example.com + modules: + tls: + - type: acme + email: joe.schmoe@example.com + challenge_type: 'http-01' + http: + - type: redirect + from: /deprecated/path + to: /new/path + - type: proxy + port: 3000 ``` @@ -403,15 +456,14 @@ See [API.md](/API.md) TODO ---- -* http - nowww module -* http - Allow match styles of `www.*`, `*`, and `*.example.com` equally -* http - redirect based on domain name (not just path) -* tcp - bind should be able to specify localhost, uniquelocal, private, or ip -* tcp - if destination host is omitted default to localhost, if dst port is missing, default to src -* sys - handle SIGHUP -* sys - `curl https://daplie.me/goldilocks | bash -s example.com` -* oauth3 - `example.com/.well-known/domains@oauth3.org/directives.json` -* oauth3 - commandline questionnaire -* modules - use consistent conventions (i.e. address vs host + port) - * tls - tls.acme vs tls.modules.acme -* tls - forward should be able to match on source port to reach different destination ports +* [ ] http - nowww module +* [ ] http - Allow match styles of `www.*`, `*`, and `*.example.com` equally +* [ ] http - redirect based on domain name (not just path) +* [ ] tcp - bind should be able to specify localhost, uniquelocal, private, or ip +* [ ] tcp - if destination host is omitted default to localhost, if dst port is missing, default to src +* [ ] sys - `curl https://daplie.me/goldilocks | bash -s example.com` +* [ ] oauth3 - `example.com/.well-known/domains@oauth3.org/directives.json` +* [ ] oauth3 - commandline questionnaire +* [x] modules - use consistent conventions (i.e. address vs host + port) + * [x] tls - tls.acme vs tls.modules.acme +* [ ] tls - forward should be able to match on source port to reach different destination ports diff --git a/bin/goldilocks.js b/bin/goldilocks.js index 4e651dc..516f6d7 100755 --- a/bin/goldilocks.js +++ b/bin/goldilocks.js @@ -8,6 +8,7 @@ if (!cluster.isMaster) { return; } +var crypto = require('crypto'); var PromiseA = require('bluebird'); var fs = PromiseA.promisifyAll(require('fs')); var configStorage; @@ -25,7 +26,158 @@ function mergeSettings(orig, changes) { } }); } -function createStorage(filename, filetype) { + +function fixRawConfig(config) { + var updated = false; + + // First converge all of the `bind` properties for protocols that are on top + // of TCP to `tcp.bind`. + if (config.tcp && config.tcp.bind && !Array.isArray(config.tcp.bind)) { + config.tcp.bind = [ config.tcp.bind ]; + updated = true; + } + if (config.http && config.http.bind) { + config.tcp = config.tcp || { bind: [] }; + config.tcp.bind = (config.tcp.bind || []).concat(config.http.bind); + delete config.http.bind; + updated = true; + } + if (config.tls && config.tls.bind) { + config.tcp = config.tcp || { bind: [] }; + config.tcp.bind = (config.tcp.bind || []).concat(config.tls.bind); + delete config.tls.bind; + updated = true; + } + + // Then we rename dns to udp since the only thing we currently do with those + // modules is proxy the packets without inspecting them at all. + if (config.dns) { + config.udp = config.dns; + delete config.dns; + updated = true; + } + // Convert all 'proxy' UDP modules to 'forward' modules that specify which + // incoming ports are relevant. Primarily to make 'proxy' modules consistent + // in needing relevant domain names. + if (config.udp && !Array.isArray(config.udp.bind)) { + config.udp.bind = [].concat(config.udp.bind || []); + updated = true; + } + if (config.udp && config.udp.modules) { + if (!config.udp.bind.length || !Array.isArray(config.udp.modules)) { + delete config.udp.modules; + updated = true; + } else { + config.udp.modules.forEach(function (mod) { + if (mod.type === 'proxy') { + mod.type = 'forward'; + mod.ports = config.udp.bind.slice(); + updated = true; + } + }); + } + } + + // This we take the old way of defining ACME options and put them into a tls module. + if (config.tls) { + var oldPropMap = { + email: 'email' + , acme_directory_url: 'server' + , challenge_type: 'challenge_type' + , servernames: 'approved_domains' + }; + if (Object.keys(oldPropMap).some(config.tls.hasOwnProperty, config.tls)) { + updated = true; + if (config.tls.acme) { + console.warn('TLS config has `acme` field and old style definitions'); + } else { + config.tls.acme = {}; + Object.keys(oldPropMap).forEach(function (oldKey) { + if (config.tls[oldKey]) { + config.tls.acme[oldPropMap[oldKey]] = config.tls[oldKey]; + } + }); + } + } + if (config.tls.acme) { + updated = true; + config.tls.acme.domains = config.tls.acme.approved_domains; + delete config.tls.acme.approved_domains; + config.tls.modules = config.tls.modules || []; + config.tls.modules.push(Object.assign({}, config.tls.acme, {type: 'acme'})); + delete config.tls.acme; + } + } + + // Then we make sure all modules have an ID and type, and makes sure all domains + // are in the right spot and also have an ID. + function updateModules(list) { + if (!Array.isArray(list)) { + return; + } + list.forEach(function (mod) { + if (!mod.id) { + mod.id = crypto.randomBytes(4).toString('hex'); + updated = true; + } + if (mod.name) { + mod.type = mod.type || mod.name; + delete mod.name; + updated = true; + } + }); + } + function moveDomains(name) { + if (!config[name].domains) { + return; + } + updated = true; + var domList = config[name].domains; + delete config[name].domains; + + if (!Array.isArray(domList)) { + return; + } + if (!Array.isArray(config.domains)) { + config.domains = []; + } + domList.forEach(function (dom) { + updateModules(dom.modules); + + var strDoms = dom.names.slice().sort().join(','); + var added = config.domains.some(function (existing) { + if (strDoms !== existing.names.slice().sort().join(',')) { + return; + } + existing.modules = existing.modules || {}; + existing.modules[name] = (existing.modules[name] || []).concat(dom.modules); + return true; + }); + if (added) { + return; + } + + var newDom = { + id: crypto.randomBytes(4).toString('hex') + , names: dom.names + , modules: {} + }; + newDom.modules[name] = dom.modules; + config.domains.push(newDom); + }); + } + + [ 'udp', 'tcp', 'http', 'tls' ].forEach(function (key) { + if (!config[key]) { + return; + } + updateModules(config[key].modules); + moveDomains(key); + }); + + return updated; +} +async function createStorage(filename, filetype) { var recase = require('recase').create({}); var snakeCopy = recase.snakeCopy.bind(recase); var camelCopy = recase.camelCopy.bind(recase); @@ -40,13 +192,25 @@ function createStorage(filename, filetype) { dump = yaml.safeDump; } - function read() { - return fs.readFileAsync(filename).then(parse).catch(function (err) { + async function read() { + var text; + try { + text = await fs.readFileAsync(filename); + } catch (err) { if (err.code === 'ENOENT') { - return ''; + return {}; } - return PromiseA.reject(err); - }); + throw err; + } + + var rawConfig = parse(text); + if (fixRawConfig(rawConfig)) { + await fs.writeFileAsync(filename, dump(rawConfig)); + text = await fs.readFileAsync(filename); + rawConfig = parse(text); + } + + return rawConfig; } var result = { @@ -76,143 +240,108 @@ function createStorage(filename, filetype) { }; return result; } -function checkConfigLocation(cwd, configFile) { +async function checkConfigLocation(cwd, configFile) { cwd = cwd || process.cwd(); var path = require('path'); - var filename; + var filename, text; - var prom; if (configFile) { filename = path.resolve(cwd, configFile); - prom = fs.readFileAsync(filename) - .catch(function (err) { - if (err.code !== 'ENOENT') { - return PromiseA.reject(err); - } - if (path.extname(filename) === '.json') { - return '{}'; - } - return ''; - }) - ; + try { + text = await fs.readFileAsync(filename); + } catch (err) { + if (err.code !== 'ENOENT') { + throw err; + } + if (path.extname(filename) === '.json') { + return { name: filename, type: 'json' }; + } else { + return { name: filename, type: 'yaml' }; + } + } } else { - prom = PromiseA.reject('blah') - .catch(function () { - filename = path.resolve(cwd, 'goldilocks.yml'); - return fs.readFileAsync(filename); - }) - .catch(function () { - filename = path.resolve(cwd, 'goldilocks.json'); - return fs.readFileAsync(filename); - }) - .catch(function () { - filename = path.resolve(cwd, 'etc/goldilocks/goldilocks.yml'); - return fs.readFileAsync(filename); - }) - .catch(function () { - filename = '/etc/goldilocks/goldilocks.yml'; - return fs.readFileAsync(filename); - }) - .catch(function () { - filename = path.resolve(cwd, 'goldilocks.yml'); - return ''; - }) - ; + // Note that `path.resolve` can handle both relative and absolute paths. + var defLocations = [ + path.resolve(cwd, 'goldilocks.yml') + , path.resolve(cwd, 'goldilocks.json') + , path.resolve(cwd, 'etc/goldilocks/goldilocks.yml') + , '/etc/goldilocks/goldilocks.yml' + ]; + + var ind; + for (ind = 0; ind < defLocations.length; ind += 1) { + try { + text = await fs.readFileAsync(defLocations[ind]); + filename = defLocations[ind]; + break; + } catch (err) { + if (err.code !== 'ENOENT') { + throw err; + } + } + } + + if (!filename) { + filename = defLocations[0]; + text = ''; + } } - return prom.then(function (text) { - try { - JSON.parse(text); - return { name: filename, type: 'json' }; - } catch (err) {} + try { + JSON.parse(text); + return { name: filename, type: 'json' }; + } catch (err) {} - try { - require('js-yaml').safeLoad(text); - return { name: filename, type: 'yaml' }; - } catch (err) {} + try { + require('js-yaml').safeLoad(text); + return { name: filename, type: 'yaml' }; + } catch (err) {} - throw new Error('Could not load "' + filename + '" as JSON nor YAML'); - }); + throw new Error('Could not load "' + filename + '" as JSON nor YAML'); } -function createConfigStorage(args) { - return checkConfigLocation(args.cwd, args.config) - .then(function (result) { - console.log('config file', result.name, 'is of type', result.type); - configStorage = createStorage(result.name, result.type); - return configStorage.read(); - }) - ; +async function createConfigStorage(args) { + var result = await checkConfigLocation(args.cwd, args.config); + console.log('config file', result.name, 'is of type', result.type); + configStorage = await createStorage(result.name, result.type); + return configStorage.read(); } var tcpProm; function fillConfig(config, args) { config.debug = config.debug || args.debug; - if (!config.dns) { - config.dns = { bind: [ 53 ], modules: [{ name: 'proxy', port: 3053 }] }; - } - if (!config.ddns) { - config.ddns = { enabled: false }; - } - // Use Object.assign to add any properties needed but not defined in the mdns config. - // It will first copy the defaults into an empty object, then copy any real config over that. - var mdnsDefaults = { port: 5353, broadcast: '224.0.0.251', ttl: 300 }; - config.mdns = Object.assign({}, mdnsDefaults, config.mdns); + config.socks5 = config.socks5 || { enabled: false }; + config.ddns = config.ddns || { enabled: false }; - if (!config.tcp) { - config.tcp = {}; - } - if (!config.http) { - config.http = { modules: [{ name: 'proxy', domains: ['*'], port: 3000 }] }; - } - if (!config.tls) { - config.tls = {}; - } - if (!config.tls.acme && (args.email || args.agreeTos)) { - config.tls.acme = {}; - } - if (typeof args.agreeTos === 'string') { - config.tls.acme.approvedDomains = args.agreeTos.split(','); - } - if (args.email) { - config.email = args.email; - config.tls.acme.email = args.email; + // Use Object.assign to copy any real config values over the default values so we can + // easily make sure all the fields we need exist . + var mdnsDefaults = { disabled: false, port: 5353, broadcast: '224.0.0.251', ttl: 300 }; + config.mdns = Object.assign(mdnsDefaults, config.mdns); + + if (!Array.isArray(config.domains)) { + config.domains = []; } - // maybe this should not go in config... but be ephemeral in some way? - config.cwd = args.cwd || config.cwd || process.cwd(); - - var ipaddr = require('ipaddr.js'); - var addresses = []; - var ifaces = require('../lib/local-ip.js').find(); - - Object.keys(ifaces).forEach(function (ifacename) { - var iface = ifaces[ifacename]; - iface.ipv4.forEach(function (ip) { - addresses.push(ip); - }); - iface.ipv6.forEach(function (ip) { - addresses.push(ip); - }); - }); - - addresses.sort(function (a, b) { - if (a.family !== b.family) { - return 'IPv4' === a.family ? 1 : -1; + function fillComponent(name, fillBind) { + if (!config[name]) { + config[name] = {}; + } + if (!Array.isArray(config[name].modules)) { + config[name].modules = []; } - return a.address > b.address ? 1 : -1; - }); + if (fillBind && !Array.isArray(config[name].bind)) { + config[name].bind = []; + } + } + fillComponent('udp', true); + fillComponent('tcp', true); + fillComponent('http', false); + fillComponent('tls', false); - addresses.forEach(function (addr) { - addr.range = ipaddr.parse(addr.address).range(); - }); - - // TODO maybe move to config.state.addresses (?) - config.addresses = addresses; config.device = { hostname: require('os').hostname() }; - if (Array.isArray(config.tcp.bind)) { + if (Array.isArray(config.tcp.bind) && config.tcp.bind.length) { return PromiseA.resolve(config); } @@ -295,6 +424,9 @@ function run(args) { // TODO spin up multiple workers // TODO use greenlock-cluster cluster.fork(); + }).catch(function (err) { + console.error(err); + process.exit(1); }) ; } @@ -308,8 +440,7 @@ function readEnv(args) { } catch (err) {} var env = { - email: process.env.GOLDILOCKS_EMAIL - , cwd: process.env.GOLDILOCKS_HOME || process.cwd() + cwd: process.env.GOLDILOCKS_HOME || process.cwd() , debug: process.env.GOLDILOCKS_DEBUG && true }; @@ -320,9 +451,7 @@ var program = require('commander'); program .version(require('../package.json').version) - .option('--agree-tos [url1,url2]', "agree to all Terms of Service for Daplie, Let's Encrypt, etc (or specific URLs only)") .option('-c --config ', 'Path to config file (Goldilocks.json or Goldilocks.yml) example: --config /etc/goldilocks/Goldilocks.json') - .option('--email ', "(Re)set default email to use for Daplie, Let's Encrypt, ACME, etc.") .option('--debug', "Enable debug output") .parse(process.argv); diff --git a/etc/goldilocks/goldilocks.example.yml b/etc/goldilocks/goldilocks.example.yml index 86ea64a..2545f4d 100644 --- a/etc/goldilocks/goldilocks.example.yml +++ b/etc/goldilocks/goldilocks.example.yml @@ -4,89 +4,92 @@ tcp: - 80 - 443 modules: - - name: forward + - type: forward ports: - 22 address: '127.0.0.1:8022' -tunnel_server: - secret: abc123 - servernames: - - 'tunnel.localhost.com' +udp: + bind: + - 53 + modules: + - type: forward + ports: + - 53 + port: 5353 + # default host is localhost + tls: - acme: - email: 'joe.shmoe@example.com' - server: 'https://acme-staging.api.letsencrypt.org/directory' - challenge_type: 'http-01' - approved_domains: - - localhost.baz.daplie.me - - localhost.beta.daplie.me - domains: - - names: - - localhost.gamma.daplie.me - modules: - - name: proxy - address: '127.0.0.1:6443' - - names: - - beta.localhost.daplie.me - - baz.localhost.daplie.me - modules: - - name: acme - email: 'owner@example.com' - challenge_type: 'tls-sni-01' - # default server is 'https://acme-v01.api.letsencrypt.org/directory' modules: - - name: proxy + - type: proxy domains: - localhost.bar.daplie.me - localhost.foo.daplie.me address: '127.0.0.1:5443' - - name: acme + - type: acme + domains: + - '*.localhost.daplie.me' email: 'guest@example.com' challenge_type: 'http-01' - domains: - - foo.localhost.daplie.me - - gamma.localhost.daplie.me - http: trust_proxy: true allow_insecure: false - primary_domain: localhost.foo.daplie.me - domains: - - names: - - localhost.baz.daplie.me - modules: - - name: redirect - from: /nowhere/in/particular - to: /just/an/example - - name: proxy - port: 3001 + primary_domain: localhost.daplie.me modules: - - name: redirect + - type: redirect domains: - localhost.beta.daplie.me status: 301 from: /old/path/*/other/* to: /path/new/:2/something/:1 - - name: proxy + - type: proxy domains: - localhost.daplie.me host: localhost port: 4000 - - name: static + - type: static domains: - '*.localhost.daplie.me' root: '/srv/www/:hostname' +domains: + - names: + - localhost.gamma.daplie.me + modules: + tls: + - type: proxy + port: 6443 + - names: + - beta.localhost.daplie.me + - baz.localhost.daplie.me + modules: + tls: + - type: acme + email: 'owner@example.com' + challenge_type: 'tls-sni-01' + # default server is 'https://acme-v01.api.letsencrypt.org/directory' + http: + - type: redirect + from: /nowhere/in/particular + to: /just/an/example + - type: proxy + address: '127.0.0.1:3001' + + mdns: disabled: false port: 5353 broadcast: '224.0.0.251' ttl: 300 +tunnel_server: + secret: abc123 + servernames: + - 'tunnel.localhost.com' + ddns: enabled: true domains: diff --git a/packages/apis/com.daplie.goldilocks/index.js b/lib/admin/apis.js similarity index 52% rename from packages/apis/com.daplie.goldilocks/index.js rename to lib/admin/apis.js index fb7f0fc..cfef538 100644 --- a/packages/apis/com.daplie.goldilocks/index.js +++ b/lib/admin/apis.js @@ -38,6 +38,13 @@ module.exports.create = function (deps, conf) { return true; } } + function makeCorsHandler(methods) { + return function corsHandler(req, res, next) { + if (!handleCors(req, res, methods)) { + next(); + } + }; + } function isAuthorized(req, res, fn) { var auth = jwt.decode((req.headers.authorization||'').replace(/^bearer\s+/i, '')); @@ -139,7 +146,10 @@ module.exports.create = function (deps, conf) { ; } - return { + // This object contains all of the API endpoints written before we changed how + // the API routing is handled. Eventually it will hopefully disappear, but for + // now we're focusing on the things that need changing more. + var oldEndPoints = { init: function (req, res) { if (handleCors(req, res, ['GET', 'POST'])) { return; @@ -235,28 +245,6 @@ module.exports.create = function (deps, conf) { }); } - , config: function (req, res) { - if (handleCors(req, res)) { - return; - } - isAuthorized(req, res, function () { - if ('POST' !== req.method) { - res.setHeader('Content-Type', 'application/json;'); - res.end(JSON.stringify(deps.recase.snakeCopy(conf))); - return; - } - - jsonParser(req, res, function () { - console.log('config POST body', req.body); - - // Since we are sending the changes to another process we don't really - // have a good way of seeing if it worked, so always report success - deps.storage.config.save(req.body); - res.setHeader('Content-Type', 'application/json;'); - res.end('{"success":true}'); - }); - }); - } , request: function (req, res) { if (handleCors(req, res, '*')) { return; @@ -331,4 +319,248 @@ module.exports.create = function (deps, conf) { }); } }; + + function handleOldApis(req, res, next) { + if (typeof oldEndPoints[req.params.name] === 'function') { + oldEndPoints[req.params.name](req, res); + } else { + next(); + } + } + + var config = { restful: {} }; + config.restful.readConfig = function (req, res, next) { + var part = new (require('./config').ConfigChanger)(conf); + if (req.params.group) { + part = part[req.params.group]; + } + if (part && req.params.domId) { + part = part.domains.findId(req.params.domId); + } + if (part && req.params.mod) { + part = part[req.params.mod]; + } + if (part && req.params.modGrp) { + part = part[req.params.modGrp]; + } + if (part && req.params.modId) { + part = part.findId(req.params.modId); + } + + if (part) { + res.send(deps.recase.snakeCopy(part)); + } else { + next(); + } + }; + + config.save = function (changer) { + var errors = changer.validate(); + if (errors.length) { + throw Object.assign(new Error(), errors[0], {statusCode: 400}); + } + + return deps.storage.config.save(changer); + }; + config.restful.saveBaseConfig = function (req, res, next) { + console.log('config POST body', JSON.stringify(req.body)); + if (req.params.group === 'domains') { + next(); + return; + } + + deps.PromiseA.resolve().then(function () { + var update; + if (req.params.group) { + update = {}; + update[req.params.group] = req.body; + } else { + update = req.body; + } + + var changer = new (require('./config').ConfigChanger)(conf); + changer.update(update); + return config.save(changer); + }).then(function (config) { + if (req.params.group) { + config = config[req.params.group]; + } + res.send(deps.recase.snakeCopy(config)); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + + config.extractModList = function (changer, params) { + var err; + if (params.domId) { + var dom = changer.domains.find(function (dom) { + return dom.id === params.domId; + }); + + if (!dom) { + err = new Error("no domain with ID '"+params.domId+"'"); + } else if (!dom.modules[params.group]) { + err = new Error("domains don't contain '"+params.group+"' modules"); + } else { + return dom.modules[params.group]; + } + } else { + if (!changer[params.group] || !changer[params.group].modules) { + err = new Error("'"+params.group+"' is not a valid settings group or doesn't support modules"); + } else { + return changer[params.group].modules; + } + } + + err.statusCode = 404; + throw err; + }; + config.restful.createModule = function (req, res, next) { + if (req.params.group === 'domains') { + next(); + return; + } + + deps.PromiseA.resolve().then(function () { + var changer = new (require('./config').ConfigChanger)(conf); + var modList = config.extractModList(changer, req.params); + + var update = req.body; + if (!Array.isArray(update)) { + update = [ update ]; + } + update.forEach(modList.add, modList); + + return config.save(changer); + }).then(function (newConf) { + res.send(deps.recase.snakeCopy(config.extractModList(newConf, req.params))); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + config.restful.updateModule = function (req, res, next) { + if (req.params.group === 'domains') { + next(); + return; + } + + deps.PromiseA.resolve().then(function () { + var changer = new (require('./config').ConfigChanger)(conf); + var modList = config.extractModList(changer, req.params); + modList.update(req.params.modId, req.body); + return config.save(changer); + }).then(function (newConf) { + res.send(deps.recase.snakeCopy(config.extractModList(newConf, req.params))); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + config.restful.removeModule = function (req, res, next) { + if (req.params.group === 'domains') { + next(); + return; + } + + deps.PromiseA.resolve().then(function () { + var changer = new (require('./config').ConfigChanger)(conf); + var modList = config.extractModList(changer, req.params); + modList.remove(req.params.modId); + return config.save(changer); + }).then(function (newConf) { + res.send(deps.recase.snakeCopy(config.extractModList(newConf, req.params))); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + + config.restful.createDomain = function (req, res) { + deps.PromiseA.resolve().then(function () { + var changer = new (require('./config').ConfigChanger)(conf); + + var update = req.body; + if (!Array.isArray(update)) { + update = [ update ]; + } + update.forEach(changer.domains.add, changer.domains); + return config.save(changer); + }).then(function (config) { + res.send(deps.recase.snakeCopy(config.domains)); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + config.restful.updateDomain = function (req, res) { + deps.PromiseA.resolve().then(function () { + if (req.body.modules) { + throw Object.assign(new Error('do not add modules with this route'), {statusCode: 400}); + } + + var changer = new (require('./config').ConfigChanger)(conf); + changer.domains.update(req.params.domId, req.body); + return config.save(changer); + }).then(function (config) { + res.send(deps.recase.snakeCopy(config.domains)); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + config.restful.removeDomain = function (req, res) { + deps.PromiseA.resolve().then(function () { + var changer = new (require('./config').ConfigChanger)(conf); + changer.domains.remove(req.params.domId); + return config.save(changer); + }).then(function (config) { + res.send(deps.recase.snakeCopy(config.domains)); + }, function (err) { + res.statusCode = err.statusCode || 500; + err.message = err.message || err.toString(); + res.end(JSON.stringify({error: {message: err.message, code: err.code}})); + }); + }; + + var app = require('express')(); + + // Handle all of the API endpoints using the old definition style, and then we can + // add middleware without worrying too much about the consequences to older code. + app.use('/:name', handleOldApis); + + app.use('/', isAuthorized, jsonParser); + + // Not all config routes support PUT or DELETE, but not worth making this more specific + app.use( '/config', makeCorsHandler(['GET', 'POST', 'PUT', 'DELETE'])); + app.get( '/config', config.restful.readConfig); + app.get( '/config/:group', config.restful.readConfig); + app.get( '/config/:group/:mod(modules)/:modId?', config.restful.readConfig); + app.get( '/config/domains/:domId/:mod(modules)?', config.restful.readConfig); + app.get( '/config/domains/:domId/:mod(modules)/:modGrp/:modId?', config.restful.readConfig); + + app.post( '/config', config.restful.saveBaseConfig); + app.post( '/config/:group', config.restful.saveBaseConfig); + + app.post( '/config/:group/modules', config.restful.createModule); + app.put( '/config/:group/modules/:modId', config.restful.updateModule); + app.delete('/config/:group/modules/:modId', config.restful.removeModule); + + app.post( '/config/domains/:domId/modules/:group', config.restful.createModule); + app.put( '/config/domains/:domId/modules/:group/:modId', config.restful.updateModule); + app.delete('/config/domains/:domId/modules/:group/:modId', config.restful.removeModule); + + app.post( '/config/domains', config.restful.createDomain); + app.put( '/config/domains/:domId', config.restful.updateDomain); + app.delete('/config/domains/:domId', config.restful.removeDomain); + + return app; }; diff --git a/lib/admin/config.js b/lib/admin/config.js new file mode 100644 index 0000000..682052c --- /dev/null +++ b/lib/admin/config.js @@ -0,0 +1,347 @@ +'use strict'; + +var validator = new (require('jsonschema').Validator)(); +var recase = require('recase').create({}); + +var portSchema = { type: 'number', minimum: 1, maximum: 65535 }; + +var moduleSchemas = { + // the proxy module is common to basically all categories. + proxy: { + type: 'object' + , oneOf: [ + { required: [ 'address' ] } + , { required: [ 'port' ] } + ] + , properties: { + address: { type: 'string' } + , host: { type: 'string' } + , port: portSchema + } + } + + // redirect and static modules are for HTTP +, redirect: { + type: 'object' + , required: [ 'to', 'from' ] + , properties: { + to: { type: 'string'} + , from: { type: 'string'} + , status: { type: 'integer', minimum: 1, maximum: 999 } + , } + } +, static: { + type: 'object' + , required: [ 'root' ] + , properties: { + root: { type: 'string' } + } + } + + // the acme module is for TLS +, acme: { + type: 'object' + , required: [ 'email' ] + , properties: { + email: { type: 'string' } + , server: { type: 'string' } + , challenge_type: { type: 'string' } + } + } +}; +// forward is basically the same as proxy, but specifies the relevant incoming port(s). +// only allows for the raw transport layers (TCP/UDP) +moduleSchemas.forward = JSON.parse(JSON.stringify(moduleSchemas.proxy)); +moduleSchemas.forward.required = [ 'ports' ]; +moduleSchemas.forward.properties.ports = { type: 'array', items: portSchema }; + +Object.keys(moduleSchemas).forEach(function (name) { + var schema = moduleSchemas[name]; + schema.id = '/modules/'+name; + schema.required = ['id', 'type'].concat(schema.required || []); + schema.properties.id = { type: 'string' }; + schema.properties.type = { type: 'string', const: name }; + validator.addSchema(schema, schema.id); +}); + +function toSchemaRef(name) { + return { '$ref': '/modules/'+name }; +} +var moduleRefs = { + http: [ 'proxy', 'static', 'redirect' ].map(toSchemaRef) +, tls: [ 'proxy', 'acme' ].map(toSchemaRef) +, tcp: [ 'forward' ].map(toSchemaRef) +, udp: [ 'forward' ].map(toSchemaRef) +}; + +function addDomainRequirement(itemSchema) { + itemSchema.required = (itemSchema.required || []).concat('domains'); + itemSchema.properties = itemSchema.properties || {}; + itemSchema.domains = { type: 'array', items: { type: 'string' }, minLength: 1}; + return itemSchema; +} + +var domainSchema = { + type: 'array' +, items: { + type: 'object' + , properties: { + id: { type: 'string' } + , names: { type: 'array', items: { type: 'string' }, minLength: 1} + , modules: { + type: 'object' + , properties: { + tls: { type: 'array', items: { oneOf: moduleRefs.tls }} + , http: { type: 'array', items: { oneOf: moduleRefs.http }} + } + , additionalProperties: false + } + } + } +}; + +var httpSchema = { + type: 'object' +, properties: { + modules: { type: 'array', items: addDomainRequirement({ oneOf: moduleRefs.http }) } + + // These properties should be snake_case to match the API and config format + , primary_domain: { type: 'string' } + , allow_insecure: { type: 'boolean' } + , trust_proxy: { type: 'boolean' } + + // these are forbidden deprecated settings. + , bind: { not: {} } + , domains: { not: {} } + } +}; + +var tlsSchema = { + type: 'object' +, properties: { + modules: { type: 'array', items: addDomainRequirement({ oneOf: moduleRefs.tls }) } + + // these are forbidden deprecated settings. + , acme: { not: {} } + , bind: { not: {} } + , domains: { not: {} } + } +}; + +var tcpSchema = { + type: 'object' +, required: [ 'bind' ] +, properties: { + bind: { type: 'array', items: portSchema, minLength: 1 } + , modules: { type: 'array', items: { oneOf: moduleRefs.tcp }} + } +}; + +var udpSchema = { + type: 'object' +, properties: { + bind: { type: 'array', items: portSchema } + , modules: { type: 'array', items: { oneOf: moduleRefs.udp }} + } +}; + +var mdnsSchema = { + type: 'object' +, required: [ 'port', 'broadcast', 'ttl' ] +, properties: { + port: portSchema + , broadcast: { type: 'string' } + , ttl: { type: 'integer', minimum: 0, maximum: 2147483647 } + } +}; + +var ddnsSchema = { + type: 'object' +, properties: { + enabled: { type: 'boolean' } + } +}; +var socks5Schema = { + type: 'object' +, properties: { + enabled: { type: 'boolean' } + , port: portSchema + } +}; +var deviceSchema = { + type: 'object' +, properties: { + hostname: { type: 'string' } + } +}; + +var mainSchema = { + type: 'object' +, required: [ 'domains', 'http', 'tls', 'tcp', 'udp', 'mdns', 'ddns' ] +, properties: { + domains:domainSchema + , http: httpSchema + , tls: tlsSchema + , tcp: tcpSchema + , udp: udpSchema + , mdns: mdnsSchema + , ddns: ddnsSchema + , socks5: socks5Schema + , device: deviceSchema + } +, additionalProperties: false +}; + +function validate(config) { + return validator.validate(recase.snakeCopy(config), mainSchema).errors; +} +module.exports.validate = validate; + +class IdList extends Array { + constructor(rawList) { + super(); + if (Array.isArray(rawList)) { + Object.assign(this, JSON.parse(JSON.stringify(rawList))); + } + this._itemName = 'item'; + } + + findId(id) { + return Array.prototype.find.call(this, function (dom) { + return dom.id === id; + }); + } + + add(item) { + item.id = require('crypto').randomBytes(4).toString('hex'); + this.push(item); + } + + update(id, update) { + var item = this.findId(id); + if (!item) { + var error = new Error("no "+this._itemName+" with ID '"+id+"'"); + error.statusCode = 404; + throw error; + } + Object.assign(this.findId(id), update); + } + + remove(id) { + var index = this.findIndex(function (dom) { + return dom.id === id; + }); + if (index < 0) { + var error = new Error("no "+this._itemName+" with ID '"+id+"'"); + error.statusCode = 404; + throw error; + } + this.splice(index, 1); + } +} +class ModuleList extends IdList { + constructor(rawList) { + super(rawList); + this._itemName = 'module'; + } + + add(mod) { + if (!mod.type) { + throw new Error("module must have a 'type' defined"); + } + if (!moduleSchemas[mod.type]) { + throw new Error("invalid module type '"+mod.type+"'"); + } + + mod.id = require('crypto').randomBytes(4).toString('hex'); + this.push(mod); + } +} +class DomainList extends IdList { + constructor(rawList) { + super(rawList); + this._itemName = 'domain'; + this.forEach(function (dom) { + dom.modules = { + http: new ModuleList((dom.modules || {}).http) + , tls: new ModuleList((dom.modules || {}).tls) + }; + }); + } + + add(dom) { + if (!Array.isArray(dom.names) || !dom.names.length) { + throw new Error("domains must have a non-empty array for 'names'"); + } + if (dom.names.some(function (name) { return typeof name !== 'string'; })) { + throw new Error("all domain names must be strings"); + } + + var modLists = { + http: new ModuleList() + , tls: new ModuleList() + }; + // We add these after instead of in the constructor to run the validation and manipulation + // in the ModList add function since these are all new modules. + if (dom.modules && Array.isArray(dom.modules.http)) { + dom.modules.http.forEach(modLists.http.add, modLists.http); + } + if (dom.modules && Array.isArray(dom.modules.tls)) { + dom.modules.tls.forEach(modLists.tls.add, modLists.tls); + } + + dom.id = require('crypto').randomBytes(4).toString('hex'); + dom.modules = modLists; + this.push(dom); + } +} + +class ConfigChanger { + constructor(start) { + Object.assign(this, JSON.parse(JSON.stringify(start))); + delete this.device; + + this.domains = new DomainList(this.domains); + this.http.modules = new ModuleList(this.http.modules); + this.tls.modules = new ModuleList(this.tls.modules); + this.tcp.modules = new ModuleList(this.tcp.modules); + this.udp.modules = new ModuleList(this.udp.modules); + } + + update(update) { + var self = this; + + if (update.domains) { + update.domains.forEach(self.domains.add, self.domains); + } + [ 'http', 'tls', 'tcp', 'udp' ].forEach(function (name) { + if (update[name] && update[name].modules) { + update[name].modules.forEach(self[name].modules.add, self[name].modules); + delete update[name].modules; + } + }); + + function mergeSettings(orig, changes) { + Object.keys(changes).forEach(function (key) { + // TODO: use an API that can properly handle updating arrays. + if (!changes[key] || (typeof changes[key] !== 'object') || Array.isArray(changes[key])) { + orig[key] = changes[key]; + } + else if (!orig[key] || typeof orig[key] !== 'object') { + orig[key] = changes[key]; + } + else { + mergeSettings(orig[key], changes[key]); + } + }); + } + mergeSettings(this, update); + + return validate(this); + } + + validate() { + return validate(this); + } +} +module.exports.ConfigChanger = ConfigChanger; diff --git a/lib/admin/index.js b/lib/admin/index.js new file mode 100644 index 0000000..e317d39 --- /dev/null +++ b/lib/admin/index.js @@ -0,0 +1,31 @@ +var adminDomains = [ + 'localhost.alpha.daplie.me' +, 'localhost.admin.daplie.me' +, 'alpha.localhost.daplie.me' +, 'admin.localhost.daplie.me' +, 'localhost.daplie.invalid' +]; +module.exports.adminDomains = adminDomains; + +module.exports.create = function (deps, conf) { + 'use strict'; + + var path = require('path'); + var express = require('express'); + var app = express(); + + var apis = require('./apis').create(deps, conf); + app.use('/api/goldilocks@daplie.com', apis); + app.use('/api/com.daplie.goldilocks', apis); + + // Serve the static assets for the UI (even though it probably won't be used very + // often since it only works on localhost domains). Note that we are using the default + // .well-known directory from the oauth3 library even though it indicates we have + // capabilities we don't support because it's simpler and it's unlikely anything will + // actually use it to determine our API (it is needed to log into the web page). + app.use('/.well-known', express.static(path.join(__dirname, '../../packages/assets/well-known'))); + app.use('/assets', express.static(path.join(__dirname, '../../packages/assets'))); + app.use('/', express.static(path.join(__dirname, '../../admin/public'))); + + return require('http').createServer(app); +}; diff --git a/lib/app.js b/lib/app.js deleted file mode 100644 index 356f0da..0000000 --- a/lib/app.js +++ /dev/null @@ -1,306 +0,0 @@ -'use strict'; - -module.exports = function (myDeps, conf, overrideHttp) { - var express = require('express'); - //var finalhandler = require('finalhandler'); - var serveStatic = require('serve-static'); - var serveIndex = require('serve-index'); - //var assetServer = serveStatic(opts.assetsPath); - var path = require('path'); - //var wellKnownServer = serveStatic(path.join(opts.assetsPath, 'well-known')); - - var serveStaticMap = {}; - var serveIndexMap = {}; - var content = conf.content; - //var server; - var goldilocksApis; - var app; - - app = express(); - - var Sites = { - add: function (sitesMap, site) { - if (!sitesMap[site.$id]) { - sitesMap[site.$id] = site; - } - - if (!site.paths) { - site.paths = []; - } - if (!site.paths._map) { - site.paths._map = {}; - } - site.paths.forEach(function (path) { - - site.paths._map[path.$id] = path; - - if (!path.modules) { - path.modules = []; - } - if (!path.modules._map) { - path.modules._map = {}; - } - path.modules.forEach(function (module) { - - path.modules._map[module.$id] = module; - }); - }); - } - }; - - var opts = overrideHttp || conf.http; - if (!opts.defaults) { - opts.defaults = {}; - } - if (!opts.global) { - opts.global = {}; - } - if (!opts.sites) { - opts.sites = []; - } - opts.sites._map = {}; - opts.sites.forEach(function (site) { - - Sites.add(opts.sites._map, site); - }); - - function mapMap(el, i, arr) { - arr._map[el.$id] = el; - } - opts.global.modules._map = {}; - opts.global.modules.forEach(mapMap); - opts.global.paths._map = {}; - opts.global.paths.forEach(function (path, i, arr) { - mapMap(path, i, arr); - //opts.global.paths._map[path.$id] = path; - path.modules._map = {}; - path.modules.forEach(mapMap); - }); - opts.sites.forEach(function (site) { - site.paths._map = {}; - site.paths.forEach(function (path, i, arr) { - mapMap(path, i, arr); - //site.paths._map[path.$id] = path; - path.modules._map = {}; - path.modules.forEach(mapMap); - }); - }); - opts.defaults.modules._map = {}; - opts.defaults.modules.forEach(mapMap); - opts.defaults.paths._map = {}; - opts.defaults.paths.forEach(function (path, i, arr) { - mapMap(path, i, arr); - //opts.global.paths._map[path.$id] = path; - path.modules._map = {}; - path.modules.forEach(mapMap); - }); - - function _goldApis(req, res, next) { - if (!goldilocksApis) { - goldilocksApis = require('../packages/apis/com.daplie.goldilocks').create(myDeps, conf); - } - - if (typeof goldilocksApis[req.params.name] === 'function') { - goldilocksApis[req.params.name](req, res); - } else { - next(); - } - } - return app - .use('/api/com.daplie.goldilocks/:name', _goldApis) - .use('/api/goldilocks@daplie.com/:name', _goldApis) - .use('/', function (req, res, next) { - if (!req.headers.host) { - next(new Error('missing HTTP Host header')); - return; - } - - if (content && '/' === req.url) { - // res.setHeader('Content-Type', 'application/octet-stream'); - res.end(content); - return; - } - - //var done = finalhandler(req, res); - var host = req.headers.host; - var hostname = (host||'').split(':')[0].toLowerCase(); - - console.log('opts.global', opts.global); - var sites = [ opts.global || null, opts.sites._map[hostname] || null, opts.defaults || null ]; - var loadables = { - serve: function (config, hostname, pathname, req, res, next) { - var originalUrl = req.url; - var dirpaths = config.paths.slice(0); - - function nextServe() { - var dirname = dirpaths.pop(); - if (!dirname) { - req.url = originalUrl; - next(); - return; - } - - console.log('[serve]', req.url, hostname, pathname, dirname); - dirname = path.resolve(conf.cwd, dirname.replace(/:hostname/, hostname)); - if (!serveStaticMap[dirname]) { - serveStaticMap[dirname] = serveStatic(dirname); - } - - serveStaticMap[dirname](req, res, nextServe); - } - - req.url = req.url.substr(pathname.length - 1); - nextServe(); - } - , indexes: function (config, hostname, pathname, req, res, next) { - var originalUrl = req.url; - var dirpaths = config.paths.slice(0); - - function nextIndex() { - var dirname = dirpaths.pop(); - if (!dirname) { - req.url = originalUrl; - next(); - return; - } - - console.log('[indexes]', req.url, hostname, pathname, dirname); - dirname = path.resolve(conf.cwd, dirname.replace(/:hostname/, hostname)); - if (!serveStaticMap[dirname]) { - serveIndexMap[dirname] = serveIndex(dirname); - } - serveIndexMap[dirname](req, res, nextIndex); - } - - req.url = req.url.substr(pathname.length - 1); - nextIndex(); - } - , app: function (config, hostname, pathname, req, res, next) { - //var appfile = path.resolve(/*process.cwd(), */config.path.replace(/:hostname/, hostname)); - var appfile = config.path.replace(/:hostname/, hostname); - try { - var app = require(appfile); - app(req, res, next); - } catch (err) { - next(); - } - } - }; - - function runModule(module, hostname, pathname, modulename, req, res, next) { - if (!loadables[modulename]) { - next(new Error("no module '" + modulename + "' found")); - return; - } - loadables[modulename](module, hostname, pathname, req, res, next); - } - - function iterModules(modules, hostname, pathname, req, res, next) { - console.log('modules'); - console.log(modules); - var modulenames = Object.keys(modules._map); - - function nextModule() { - var modulename = modulenames.pop(); - if (!modulename) { - next(); - return; - } - - console.log('modules', modules); - runModule(modules._map[modulename], hostname, pathname, modulename, req, res, nextModule); - } - - nextModule(); - } - - function iterPaths(site, hostname, req, res, next) { - console.log('site', hostname); - console.log(site); - var pathnames = Object.keys(site.paths._map); - console.log('pathnames', pathnames); - pathnames = pathnames.filter(function (pathname) { - // TODO ensure that pathname has trailing / - return (0 === req.url.indexOf(pathname)); - //return req.url.match(pathname); - }); - pathnames.sort(function (a, b) { - return b.length - a.length; - }); - console.log('pathnames', pathnames); - - function nextPath() { - var pathname = pathnames.shift(); - if (!pathname) { - next(); - return; - } - - console.log('iterPaths', hostname, pathname, req.url); - iterModules(site.paths._map[pathname].modules, hostname, pathname, req, res, nextPath); - } - - nextPath(); - } - - function nextSite() { - console.log('hostname', hostname, sites); - var site; - if (!sites.length) { - next(); // 404 - return; - } - site = sites.shift(); - if (!site) { - nextSite(); - return; - } - iterPaths(site, hostname, req, res, nextSite); - } - - nextSite(); - - /* - function serveStaticly(server) { - function serveTheStatic() { - server.serve(req, res, function (err) { - if (err) { return done(err); } - server.index(req, res, function (err) { - if (err) { return done(err); } - req.url = req.url.replace(/\/assets/, ''); - assetServer(req, res, function () { - if (err) { return done(err); } - req.url = req.url.replace(/\/\.well-known/, ''); - wellKnownServer(req, res, done); - }); - }); - }); - } - - if (server.expressApp) { - server.expressApp(req, res, serveTheStatic); - return; - } - - serveTheStatic(); - } - - if (opts.livereload) { - res.__my_livereload = ''; - res.__my_addLen = res.__my_livereload.length; - - // TODO modify prototype instead of each instance? - res.__write = res.write; - res.write = _reloadWrite; - } - - console.log('hostname:', hostname, opts.sites[0].paths); - - addServer(hostname); - server = hostsMap[hostname] || hostsMap[opts.sites[0].name]; - serveStaticly(server); - */ - }); -}; diff --git a/lib/goldilocks.js b/lib/goldilocks.js index a0b3fcc..8ad86b3 100644 --- a/lib/goldilocks.js +++ b/lib/goldilocks.js @@ -95,16 +95,20 @@ module.exports.create = function (deps, config) { }); } - function dnsListener(msg) { - if (!Array.isArray(config.dns.modules)) { + function dnsListener(port, msg) { + if (!Array.isArray(config.udp.modules)) { return; } var socket = require('dgram').createSocket('udp4'); - config.dns.modules.forEach(function (mod) { - if (mod.name !== 'proxy') { + config.udp.modules.forEach(function (mod) { + if (mod.type !== 'forward') { console.warn('found bad DNS module', mod); return; } + if (mod.ports.indexOf(port) < 0) { + return; + } + var dest = require('./domain-utils').separatePort(mod.address || ''); dest.port = dest.port || mod.port; dest.host = dest.host || mod.host || 'localhost'; @@ -197,23 +201,12 @@ module.exports.create = function (deps, config) { var listenPromises = []; var tcpPortMap = {}; - function addPorts(bindList) { - if (!bindList) { - return; - } - if (Array.isArray(bindList)) { - bindList.filter(Number).forEach(function (port) { - tcpPortMap[port] = true; - }); - } - else if (Number(bindList)) { - tcpPortMap[bindList] = true; - } - } + config.tcp.bind.filter(Number).forEach(function (port) { + tcpPortMap[port] = true; + }); - addPorts(config.tcp.bind); (config.tcp.modules || []).forEach(function (mod) { - if (mod.name === 'forward') { + if (mod.type === 'forward') { var forwarder = createTcpForwarder(mod); mod.ports.forEach(function (port) { if (!tcpPortMap[port]) { @@ -229,25 +222,15 @@ module.exports.create = function (deps, config) { } }); - // Even though these ports were specified in different places we treat any TCP - // connections we haven't been told to just forward exactly as is equal so that - // we can potentially use the same ports for different protocols. - addPorts(config.tls.bind); - addPorts(config.http.bind); - var portList = Object.keys(tcpPortMap).map(Number).sort(); portList.forEach(function (port) { listenPromises.push(listeners.tcp.add(port, netHandler)); }); - if (config.dns.bind) { - if (Array.isArray(config.dns.bind)) { - config.dns.bind.map(function (port) { - listenPromises.push(listeners.udp.add(port, dnsListener)); - }); - } else { - listenPromises.push(listeners.udp.add(config.dns.bind, dnsListener)); - } + if (config.udp.bind) { + config.udp.bind.forEach(function (port) { + listenPromises.push(listeners.udp.add(port, dnsListener.bind(port))); + }); } if (!config.mdns.disabled) { diff --git a/lib/modules/admin.js b/lib/modules/admin.js deleted file mode 100644 index d22b168..0000000 --- a/lib/modules/admin.js +++ /dev/null @@ -1,67 +0,0 @@ -var adminDomains = [ - 'localhost.alpha.daplie.me' -, 'localhost.admin.daplie.me' -, 'alpha.localhost.daplie.me' -, 'admin.localhost.daplie.me' -, 'localhost.daplie.invalid' -]; -module.exports.adminDomains = adminDomains; - -module.exports.create = function (deps, conf) { - 'use strict'; - - var path = require('path'); - //var defaultServername = 'localhost.daplie.me'; - //var defaultWebRoot = '.'; - var assetsPath = path.join(__dirname, '..', '..', 'packages', 'assets'); - var opts = {}; - - opts.global = opts.global || {}; - opts.sites = opts.sites || []; - opts.sites._map = {}; - - // argv.sites - - opts.groups = []; - - // 'packages', 'assets', 'com.daplie.goldilocks' - opts.global = { - modules: [ // TODO uh-oh we've got a mixed bag of modules (various types), a true map - { $id: 'greenlock', email: opts.email, tos: opts.tos } - , { $id: 'rvpn', email: opts.email, tos: opts.tos } - //, { $id: 'content', content: content } - , { $id: 'livereload', on: opts.livereload } - , { $id: 'app', path: opts.expressApp } - ] - , paths: [ - { $id: '/assets/', modules: [ { $id: 'serve', paths: [ assetsPath ] } ] } - // TODO figure this b out - , { $id: '/.well-known/', modules: [ - { $id: 'serve', paths: [ path.join(assetsPath, 'well-known') ] } - ] } - ] - }; - opts.defaults = { - modules: [] - , paths: [ - /* - { $id: '/', modules: [ - { $id: 'serve', paths: [ defaultWebRoot ] } - , { $id: 'indexes', paths: [ defaultWebRoot ] } - ] } - */ - ] - }; - adminDomains.forEach(function (id) { - opts.sites.push({ - $id: id - , paths: [ - { $id: '/', modules: [ { $id: 'serve', paths: [ path.resolve(__dirname, '..', '..', 'admin', 'public') ] } ] } - , { $id: '/api/', modules: [ { $id: 'app', path: path.join(__dirname, 'admin') } ] } - ] - }); - }); - - var app = require('../app.js')(deps, conf, opts); - return require('http').createServer(app); -}; diff --git a/lib/modules/http.js b/lib/modules/http.js index 3b94d46..7f615f6 100644 --- a/lib/modules/http.js +++ b/lib/modules/http.js @@ -65,18 +65,21 @@ module.exports.create = function (deps, conf, greenlockMiddleware) { }); } - function hostMatchesDomains(req, domains) { + function hostMatchesDomains(req, domainList) { var host = separatePort((req.headers || req).host).host.toLowerCase(); - return domains.some(function (pattern) { + return domainList.some(function (pattern) { return domainMatches(pattern, host); }); } function determinePrimaryHost() { var result; - if (Array.isArray(conf.http.domains)) { - conf.http.domains.some(function (dom) { + if (Array.isArray(conf.domains)) { + conf.domains.some(function (dom) { + if (!dom.modules || !dom.modules.http) { + return false; + } return dom.names.some(function (domain) { if (domain[0] !== '*') { result = domain; @@ -202,11 +205,11 @@ module.exports.create = function (deps, conf, greenlockMiddleware) { var host = separatePort(headers.host).host; if (!adminDomains) { - adminDomains = require('./admin').adminDomains; + adminDomains = require('../admin').adminDomains; } if (adminDomains.indexOf(host) !== -1) { if (!adminServer) { - adminServer = require('./admin').create(deps, conf); + adminServer = require('../admin').create(deps, conf); } return emitConnection(adminServer, conn, opts); } @@ -415,21 +418,24 @@ module.exports.create = function (deps, conf, greenlockMiddleware) { if (checkAdmin(conn, opts, headers)) { return; } var prom = PromiseA.resolve(false); - (conf.http.domains || []).forEach(function (dom) { + (conf.domains || []).forEach(function (dom) { prom = prom.then(function (handled) { if (handled) { return handled; } + if (!dom.modules || !dom.modules.http) { + return false; + } if (!hostMatchesDomains(headers, dom.names)) { return false; } var subProm = PromiseA.resolve(false); - dom.modules.forEach(function (mod) { - if (moduleChecks[mod.name]) { + dom.modules.http.forEach(function (mod) { + if (moduleChecks[mod.type]) { subProm = subProm.then(function (handled) { if (handled) { return handled; } - return moduleChecks[mod.name](mod, conn, opts, headers); + return moduleChecks[mod.type](mod, conn, opts, headers); }); } else { console.warn('unknown HTTP module under domains', dom.names.join(','), mod); @@ -447,8 +453,8 @@ module.exports.create = function (deps, conf, greenlockMiddleware) { return false; } - if (moduleChecks[mod.name]) { - return moduleChecks[mod.name](mod, conn, opts, headers); + if (moduleChecks[mod.type]) { + return moduleChecks[mod.type](mod, conn, opts, headers); } console.warn('unknown HTTP module found', mod); }); diff --git a/lib/modules/tls.js b/lib/modules/tls.js index 19bb2f1..2b9a614 100644 --- a/lib/modules/tls.js +++ b/lib/modules/tls.js @@ -27,8 +27,8 @@ module.exports.create = function (deps, config, netHandler) { return value || ''; } - function nameMatchesDomains(name, domains) { - return domains.some(function (pattern) { + function nameMatchesDomains(name, domainList) { + return domainList.some(function (pattern) { return domainMatches(pattern, name); }); } @@ -135,14 +135,17 @@ module.exports.create = function (deps, config, netHandler) { } var handled = false; - if (Array.isArray(config.tls.domains)) { - handled = config.tls.domains.some(function (dom) { + if (Array.isArray(config.domains)) { + handled = config.domains.some(function (dom) { + if (!dom.modules || !dom.modules.tls) { + return false; + } if (!nameMatchesDomains(opts.domain, dom.names)) { return false; } - return dom.modules.some(function (mod) { - if (mod.name !== 'acme') { + return dom.modules.tls.some(function (mod) { + if (mod.type !== 'acme') { return false; } complete(mod, dom.names); @@ -156,7 +159,7 @@ module.exports.create = function (deps, config, netHandler) { if (Array.isArray(config.tls.modules)) { handled = config.tls.modules.some(function (mod) { - if (mod.name !== 'acme') { + if (mod.type !== 'acme') { return false; } if (!nameMatchesDomains(opts.domain, mod.domains)) { @@ -171,26 +174,6 @@ module.exports.create = function (deps, config, netHandler) { return; } - var defAcmeConf; - if (config.tls.acme) { - defAcmeConf = config.tls.acme; - } else { - defAcmeConf = { - email: config.tls.email - , server: config.tls.acmeDirectoryUrl || le.server - , challengeType: config.tls.challengeType || le.challengeType - , approvedDomains: config.tls.servernames - }; - } - - // Check config for domain name - // TODO: if `approvedDomains` isn't defined check all other modules to see if they can - // handle this domain (and what other domains it's grouped with). - if (-1 !== (defAcmeConf.approvedDomains || []).indexOf(opts.domain)) { - complete(defAcmeConf, defAcmeConf.approvedDomains); - return; - } - cb(new Error('domain is not allowed')); } }); @@ -322,20 +305,23 @@ module.exports.create = function (deps, config, netHandler) { } function checkModule(mod) { - if (mod.name === 'proxy') { + if (mod.type === 'proxy') { return proxy(socket, opts, mod); } - if (mod.name !== 'acme') { + if (mod.type !== 'acme') { console.error('saw unknown TLS module', mod); } } - var handled = (config.tls.domains || []).some(function (dom) { + var handled = (config.domains || []).some(function (dom) { + if (!dom.modules || !dom.modules.tls) { + return false; + } if (!nameMatchesDomains(opts.servername, dom.names)) { return false; } - return dom.modules.some(checkModule); + return dom.modules.tls.some(checkModule); }); if (handled) { return; diff --git a/lib/storage.js b/lib/storage.js index 4651f2d..33fc8f6 100644 --- a/lib/storage.js +++ b/lib/storage.js @@ -65,14 +65,33 @@ module.exports.create = function (deps, conf) { } }; + var confCb; var config = { save: function (changes) { deps.messenger.send({ type: 'com.daplie.goldilocks/config' , changes: changes }); + + return new deps.PromiseA(function (resolve, reject) { + var timeoutId = setTimeout(function () { + reject(new Error('Did not receive config update from main process in a reasonable time')); + confCb = null; + }, 15*1000); + + confCb = function (config) { + confCb = null; + clearTimeout(timeoutId); + resolve(config); + }; + }); } }; + function updateConf(config) { + if (confCb) { + confCb(config); + } + } var mdnsId = { _filename: 'mdns-id' @@ -99,6 +118,7 @@ module.exports.create = function (deps, conf) { return { owners: owners , config: config + , updateConf: updateConf , mdnsId: mdnsId }; }; diff --git a/lib/worker.js b/lib/worker.js index 44f4507..665c48e 100644 --- a/lib/worker.js +++ b/lib/worker.js @@ -1,6 +1,7 @@ 'use strict'; var config; +var modules; // Everything that uses the config should be reading it when relevant rather than // just at the beginning, so we keep the reference for the main object and just @@ -15,7 +16,13 @@ function update(conf) { config[key] = conf[key]; } }); - console.log('config', JSON.stringify(config)); + + console.log('config update', JSON.stringify(config)); + Object.values(modules).forEach(function (mod) { + if (typeof mod.updateConf === 'function') { + mod.updateConf(config); + } + }); } function create(conf) { @@ -38,10 +45,14 @@ function create(conf) { // HTTP proxying connection creation is not something we currently control. , net: require('net') }; - deps.storage = require('./storage').create(deps, conf); - deps.proxy = require('./proxy-conn').create(deps, conf); - deps.socks5 = require('./socks5-server').create(deps, conf); - deps.ddns = require('./ddns').create(deps, conf); + + modules = { + storage: require('./storage').create(deps, conf) + , proxy: require('./proxy-conn').create(deps, conf) + , socks5: require('./socks5-server').create(deps, conf) + , ddns: require('./ddns').create(deps, conf) + }; + Object.assign(deps, modules); require('./goldilocks.js').create(deps, conf); process.removeListener('message', create); diff --git a/package-lock.json b/package-lock.json index 02dd078..49f699d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1029,6 +1029,11 @@ "resolved": "https://registry.npmjs.org/jsonify/-/jsonify-0.0.0.tgz", "integrity": "sha1-LHS27kHZPKUbe1qu6PUDYx0lKnM=" }, + "jsonschema": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/jsonschema/-/jsonschema-1.2.0.tgz", + "integrity": "sha512-XDJApzBauMg0TinJNP4iVcJl99PQ4JbWKK7nwzpOIkAOVveDKgh/2xm41T3x7Spu4PWMhnnQpNJmUSIUgl6sKg==" + }, "jsonwebtoken": { "version": "7.4.1", "resolved": "https://registry.npmjs.org/jsonwebtoken/-/jsonwebtoken-7.4.1.tgz", diff --git a/package.json b/package.json index e026a14..1d70e3f 100644 --- a/package.json +++ b/package.json @@ -49,6 +49,7 @@ "human-readable-ids": "git+https://git.daplie.com/Daplie/human-readable-ids-js#master", "ipaddr.js": "git+https://github.com/whitequark/ipaddr.js.git#v1.3.0", "js-yaml": "^3.8.3", + "jsonschema": "^1.2.0", "jsonwebtoken": "^7.4.0", "le-challenge-ddns": "git+https://git.daplie.com/Daplie/le-challenge-ddns.git#master", "le-challenge-fs": "git+https://git.daplie.com/Daplie/le-challenge-webroot.git#master", diff --git a/packages/apis/com.daplie.goldilocks/test.js b/packages/apis/com.daplie.goldilocks/test.js deleted file mode 100644 index 77b55de..0000000 --- a/packages/apis/com.daplie.goldilocks/test.js +++ /dev/null @@ -1,23 +0,0 @@ -'use strict'; - -var api = require('./index.js').api; -var OAUTH3 = require('../../assets/org.oauth3/'); -// these all auto-register -require('../../assets/org.oauth3/oauth3.domains.js'); -require('../../assets/org.oauth3/oauth3.dns.js'); -require('../../assets/org.oauth3/oauth3.tunnel.js'); -OAUTH3._hooks = require('../../assets/org.oauth3/oauth3.node.storage.js'); - -api.tunnel( - { - OAUTH3: OAUTH3 - , options: { - device: { - hostname: 'test.local' - , id: '' - } - } - } - // OAUTH3.hooks.session.get('oauth3.org').then(function (result) { console.log(result) }); -, require('./test.session.json') -);