Wanted: sequelize plugin #1

New Issue

Open

opened 2018-10-02 23:09:31 +00:00 by coolaj86 · 3 comments

coolaj86 commented 2018-10-02 23:09:31 +00:00

Owner

Copy Link

Right now all certificates are saved to disk.

This doesn't work well for ephemeral systems that shouldn't rely on disk storage - such as AWS.

I'd like someone to help in creating some plugins (i.e. le-store-sql) that work with PostgreSQL, SQLite, MongoDB, etc through common node db adapters.

• Reference Implementation (in-memory) le-store-SPEC
• SQL le-store-sql https://github.com/Daplie/le-store-sql/issues/1
  • PostgreSQL
  • SQLite
  • MySQL
• Key / Value & Graph
  • CouchDB
  • GunDB @amark @metasean
  • MongoDB le-store-mongo
  • Redis
  • RethinkDB
• plain old JSON
• certbot (backwards compat with python config): le-store-certbot

Here's a template for what needs to be implemented:

'use strict';

module.exports.create = function (options) {



  var defaults = {};



  var accounts = {
    checkKeypair: function (opts, cb) {
      // opts.email // optional
      // opts.accountId // optional

      // check db and return null or keypair object with one of privateKeyPem or privateKeyJwk
      cb(null, { privateKeyPem: '...', privateKeyJwk: {} });
    }
  , setKeypair: function (opts, keypair, cb) {
      // opts.email // optional
      // opts.accountId // optional

      // SAVE to db (as PEM and/or JWK) and index each domain in domains to this keypair
      cb(null, keypair);
    }
  , check: function (opts, cb) {
      // opts.email // optional
      // opts.accountId // optional
      // opts.domains // optional

      // return account from db if it exists, otherwise null
      cb(null, { id: '...', keypair: { privateKeyJwk: {} }, domains: [] });
    }
  , set: function (opts, reg, cb) {
      // opts.email
      // reg.keypair
      // reg.receipt // response from acme server


      cb(null, { id: '...', email: opts.email, keypair: reg.keypair, receipt: reg.receipt });
    }
  };



  var certificates = {
    checkKeypair: function (opts, cb) {
      // opts.domains

      // check db and return null or keypair object with one of privateKeyPem or privateKeyJwk
      cb(null, { privateKeyPem: '...', privateKeyJwk: {} });
    }
  , setKeypair: function (opts, keypair, cb) {
      // opts.domains

      // SAVE to db (as PEM and/or JWK) and index each domain in domains to this keypair
      cb(null, keypair);
    }
  , check: function (opts, cb) {
      // You will be provided one of these (which should be tried in this order)
      // opts.domains
      // opts.email // optional
      // opts.accountId // optional

      // return certificate PEMs from db if they exist, otherwise null
      // optionally include expiresAt and issuedAt, if they are known exactly
      // (otherwise they will be read from the cert itself later)
      cb(null, { privkey: 'PEM', cert: 'PEM', chain: 'PEM', domains: [], accountId: '...' });
    }
  , set: function (opts, pems, cb) {
      // opts.domains
      // opts.email // optional
      // opts.accountId // optional

      // pems.privkey
      // pems.cert
      // pems.chain

      // SAVE to the database, index the email address, the accountId, and alias the domains
      cb(null, pems);
    }
  };



  return {
    getOptions: function () {
      // merge options with default settings and then return them
      return options;
    }
  , accounts: accounts
  , certificates: certificates
  };



};

Note that certificates may have up to 100 domains listed, but renewals happen in a one-off fashion, so it is required to be able to look up any certificate and any account based on any domain it is associated with and to be able to return all domains for which a certificate or account is valid.

Here's the current plugin: https://github.com/Daplie/le-store-certbot
(it's overly complicated because it's taken from the codebase that was used to maintain compatibility with the python certbot, but you can see that the 9 essential methods are implemented)

Right now all certificates are saved to disk. This doesn't work well for ephemeral systems that shouldn't rely on disk storage - such as AWS. I'd like someone to help in creating some plugins (i.e. `le-store-sql`) that work with PostgreSQL, SQLite, MongoDB, etc through common node db adapters. - [ ] Reference Implementation (in-memory) [`le-store-SPEC`](https://github.com/Daplie/le-store-SPEC) - [ ] SQL [`le-store-sql`](https://github.com/Daplie/le-store-sql) https://github.com/Daplie/le-store-sql/issues/1 - [ ] PostgreSQL - [ ] SQLite - [ ] MySQL - [ ] Key / Value & Graph - [ ] CouchDB - [ ] GunDB @amark @metasean - [ ] MongoDB [`le-store-mongo`](https://github.com/Daplie/le-store-mongo) - [ ] Redis - [ ] RethinkDB - [ ] plain old JSON - [x] certbot (backwards compat with python config): [le-store-certbot](https://github.com/Daplie/le-store-certbot) Here's a template for what needs to be implemented: ``` js 'use strict'; module.exports.create = function (options) { var defaults = {}; var accounts = { checkKeypair: function (opts, cb) { // opts.email // optional // opts.accountId // optional // check db and return null or keypair object with one of privateKeyPem or privateKeyJwk cb(null, { privateKeyPem: '...', privateKeyJwk: {} }); } , setKeypair: function (opts, keypair, cb) { // opts.email // optional // opts.accountId // optional // SAVE to db (as PEM and/or JWK) and index each domain in domains to this keypair cb(null, keypair); } , check: function (opts, cb) { // opts.email // optional // opts.accountId // optional // opts.domains // optional // return account from db if it exists, otherwise null cb(null, { id: '...', keypair: { privateKeyJwk: {} }, domains: [] }); } , set: function (opts, reg, cb) { // opts.email // reg.keypair // reg.receipt // response from acme server cb(null, { id: '...', email: opts.email, keypair: reg.keypair, receipt: reg.receipt }); } }; var certificates = { checkKeypair: function (opts, cb) { // opts.domains // check db and return null or keypair object with one of privateKeyPem or privateKeyJwk cb(null, { privateKeyPem: '...', privateKeyJwk: {} }); } , setKeypair: function (opts, keypair, cb) { // opts.domains // SAVE to db (as PEM and/or JWK) and index each domain in domains to this keypair cb(null, keypair); } , check: function (opts, cb) { // You will be provided one of these (which should be tried in this order) // opts.domains // opts.email // optional // opts.accountId // optional // return certificate PEMs from db if they exist, otherwise null // optionally include expiresAt and issuedAt, if they are known exactly // (otherwise they will be read from the cert itself later) cb(null, { privkey: 'PEM', cert: 'PEM', chain: 'PEM', domains: [], accountId: '...' }); } , set: function (opts, pems, cb) { // opts.domains // opts.email // optional // opts.accountId // optional // pems.privkey // pems.cert // pems.chain // SAVE to the database, index the email address, the accountId, and alias the domains cb(null, pems); } }; return { getOptions: function () { // merge options with default settings and then return them return options; } , accounts: accounts , certificates: certificates }; }; ``` Note that certificates may have up to 100 domains listed, but renewals happen in a one-off fashion, so it is required to be able to look up any certificate and any account based on any domain it is associated with and to be able to return all domains for which a certificate or account is valid. Here's the current plugin: https://github.com/Daplie/le-store-certbot (it's overly complicated because it's taken from the codebase that was used to maintain compatibility with the python certbot, but you can see that the 9 essential methods are implemented)

coolaj86 commented 2018-10-02 23:12:01 +00:00

Author

Owner

Copy Link

In v2.0.0, this is how you pass a custom handler:

'use strict';

var LE = require('letsencrypt');
var le;


// Storage Backend
var leStore = require('le-store-certbot').create({
  configDir: '~/letsencrypt/etc'                          // or /etc/letsencrypt or wherever
, debug: false
});


// ACME Challenge Handlers
var leChallenge = require('le-challenge-fs').create({
  webrootPath: '~/letsencrypt/var/'                       // or template string such as
, debug: false                                            // '/srv/www/:hostname/.well-known/acme-challenge'
});


function leAgree(opts, agreeCb) {
  // opts = { email, domains, tosUrl }
  agreeCb(null, opts.tosUrl);
}

le = LE.create({
  server: LE.stagingServerUrl                             // or LE.productionServerUrl
, store: leStore                                          // handles saving of config, accounts, and certificates
, challenge: leChallenge                                  // handles /.well-known/acme-challege keys and tokens
, agreeToTerms: leAgree                                   // hook to allow user to view and accept LE TOS
, debug: false
});

See https://git.coolaj86.com/coolaj86/greenlock.js

Any user can therefore create their set of handlers and then publish them to npm as le-store-* or le-challenge-* and others can use them.

In v2.0.0, this is how you pass a custom handler: ``` javascript 'use strict'; var LE = require('letsencrypt'); var le; // Storage Backend var leStore = require('le-store-certbot').create({ configDir: '~/letsencrypt/etc' // or /etc/letsencrypt or wherever , debug: false }); // ACME Challenge Handlers var leChallenge = require('le-challenge-fs').create({ webrootPath: '~/letsencrypt/var/' // or template string such as , debug: false // '/srv/www/:hostname/.well-known/acme-challenge' }); function leAgree(opts, agreeCb) { // opts = { email, domains, tosUrl } agreeCb(null, opts.tosUrl); } le = LE.create({ server: LE.stagingServerUrl // or LE.productionServerUrl , store: leStore // handles saving of config, accounts, and certificates , challenge: leChallenge // handles /.well-known/acme-challege keys and tokens , agreeToTerms: leAgree // hook to allow user to view and accept LE TOS , debug: false }); ``` See https://git.coolaj86.com/coolaj86/greenlock.js Any user can therefore create their set of handlers and then publish them to npm as `le-store-*` or `le-challenge-*` and others can use them.

coolaj86 commented 2018-10-02 23:16:13 +00:00

Author

Owner

Copy Link

Many people will have very specific requirements with very special tables, but for a great many people having some intuitively named tables will be good enough.

The problem that we're solving for is that on various types of cloud systems the filesystems are ephemeral and the database must go on a separate machine or service.

For those people, the plugins will get them up and running with their preferred cloud service and their preferred database supported by that cloud service.

For people who are not those people, it gives them a good base to start from - code to look at and modify.

@Rush Not that you need to do it my way, but I like the convention of join tables being named after what they join, alphabetically, table names being pluralized, all lowercase with underscores (no unexpected behaviors with case-sensitive filesystems like MySQL 4 had), and ids singular and having a trailing _id, and timestamps to have a trailing _at.

For example:

# regular tables

accounts
  - id STRING           # sha256 of public key
  - private_key_pem     # private key in pem format (as opposed to jwk)
  - email STRING
  - tos_url STRING      # null, 'true', or 'https://acme.example.org/tos/2016-06-16.html'

certificates
  - id STRING           # sha256 of public key
  - private_key_pem
  - subject
  - expires_at
  - issued_at



# join tables

accounts_certificates
  - account_id
  - domain_id

certificates_domains
  - certificate_id
  - domain_id STRING    # name of domain example.com or www.example.com, etc
  - subject BOOLEAN     # is the primary domain listed on the certificate


# food for thought:
# perhaps `keys` could be its own table instead of included in the accounts and certificates tables

keypairs
  - id STRING           # sha256 of public
  - private_jwk JSON    # or pem, whatever
  - size INT            # bits, i.e. 2048

Many people will have very specific requirements with very special tables, but for a great many people having some intuitively named tables will be good enough. The problem that we're solving for is that on various types of cloud systems the **filesystems are ephemeral** and the database must go on a **separate machine or service**. For those people, the plugins will get them up and running with their preferred cloud service and their preferred database supported by that cloud service. For people who are not those people, it gives them a good base to start from - code to look at and modify. @Rush Not that you need to do it my way, but I like the convention of join tables being named after what they join, alphabetically, table names being pluralized, all lowercase with underscores (no unexpected behaviors with case-sensitive filesystems like MySQL 4 had), and ids singular and having a trailing `_id`, and timestamps to have a trailing `_at`. For example: ``` # regular tables accounts - id STRING # sha256 of public key - private_key_pem # private key in pem format (as opposed to jwk) - email STRING - tos_url STRING # null, 'true', or 'https://acme.example.org/tos/2016-06-16.html' certificates - id STRING # sha256 of public key - private_key_pem - subject - expires_at - issued_at # join tables accounts_certificates - account_id - domain_id certificates_domains - certificate_id - domain_id STRING # name of domain example.com or www.example.com, etc - subject BOOLEAN # is the primary domain listed on the certificate # food for thought: # perhaps `keys` could be its own table instead of included in the accounts and certificates tables keypairs - id STRING # sha256 of public - private_jwk JSON # or pem, whatever - size INT # bits, i.e. 2048 ```

coolaj86 commented 2018-10-02 23:17:25 +00:00

Author

Owner

Copy Link

8 check / set methods

They're all listed here:

https://git.coolaj86.com/coolaj86/le-store-SPEC.js

And they're stubbed out here:

https://git.coolaj86.com/coolaj86/le-store-SPEC.js/src/branch/template/index.js

check vs get vs set: There are some abstractions higher up that are called get which are really getOrCreate, so I called these check to disambiguate.

callback order

The callback order is in the same as they're listed in the documentation:

1. accounts.checkKeypair
2. accounts.setKeypair (first time only)
3. accounts.check
4. accounts.set (first time only)
5. certificates.checkKeypair
6. certificates.setKeypair (first time only)
7. certificates.check
8. certificates.set (every renewal)

They're individually documented in the stub:
https://git.coolaj86.com/coolaj86/le-store-SPEC.js/src/branch/template/index.js

The reason the check/set keypairs are separate is twofold:

1. Debugging. Since keypairs can be created independently of a successful account or certificate registration, it makes life easier to have them separate
2. Databases: Depending on your schema you may wish to store keypairs directly in the account data or separately in their own collection or on disk.

ACME Challenge Protocols

The 3 supported protocols are http-01, tls-sni-01 (https), and dns-01. I'd recommend sticking with http-01, which is the default and uses http on port 80 for now.

I think tls-sni-01 requires a valid certificate, which could get you into trouble.

There are two implementations for dns-01. Ours, of course, requires Daplie Domains / Daplie DNS. Another guy has one for cloudflare I think.

And if you npm install from master (I haven't published v2.1.8 yet), you can actually choose which strategy to use by setting challengeType in the approveDomains callback.

When will new certs be used?

If you arbitrarily register and or renew a certificate, it will be used when the one that exists in-memory would have been renewed (because it will always call certificates.check to see if a newer cert already exists before renewing), which is randomly set between 3 and 10 days before expiration on each certificate load.

Help me

Please open a pull request on the docs with the clarifications you'd like to see added:
https://git.coolaj86.com/coolaj86/le-store-SPEC.js

## 8 check / set methods They're all listed here: https://git.coolaj86.com/coolaj86/le-store-SPEC.js And they're stubbed out here: https://git.coolaj86.com/coolaj86/le-store-SPEC.js/src/branch/template/index.js `check` vs `get` vs `set`: There are some abstractions higher up that are called `get` which are really `getOrCreate`, so I called these `check` to disambiguate. ## callback order The callback order is in the same as they're listed in the documentation: 1. accounts.checkKeypair 2. accounts.setKeypair (first time only) 3. accounts.check 4. accounts.set (first time only) 5. certificates.checkKeypair 6. certificates.setKeypair (first time only) 7. certificates.check 8. certificates.set (every renewal) They're individually documented in the stub: https://git.coolaj86.com/coolaj86/le-store-SPEC.js/src/branch/template/index.js The reason the check/set keypairs are separate is twofold: 1. Debugging. Since keypairs can be created independently of a successful account or certificate registration, it makes life easier to have them separate 2. Databases: Depending on your schema you may wish to store keypairs directly in the account data or separately in their own collection or on disk. ## ACME Challenge Protocols The 3 supported protocols are `http-01`, `tls-sni-01` (**https**), and `dns-01`. I'd recommend sticking with `http-01`, which is the default and uses http on port 80 for now. I think `tls-sni-01` requires a valid certificate, which could get you into trouble. There are two implementations for `dns-01`. Ours, of course, requires Daplie Domains / Daplie DNS. Another guy has one for cloudflare I think. And if you npm install from master (I haven't published v2.1.8 yet), you can actually choose which strategy to use by setting `challengeType` in the `approveDomains` callback. ## When will new certs be used? If you arbitrarily register and or renew a certificate, it will be used when the one that exists in-memory would have been renewed (because it will always call certificates.check to see if a newer cert already exists before renewing), which is randomly set between 3 and 10 days before expiration on each certificate load. ## Help me Please open a pull request on the docs with the clarifications you'd like to see added: https://git.coolaj86.com/coolaj86/le-store-SPEC.js

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

master

template

tests

readme

Labels

Clear labels

No items

No Label

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coolaj86/le-store-SPEC.js#1

No description provided.