In-progress changes to config and tasks

As an experiment, for the past few weeks I’ve been working on replacing bb tasks with plain clj. In the course of doing that I thought I’d also experiment with redoing the config system. I haven’t made a final decision about putting these changes into Biff, but I’m feeling good about them so far.

For the tasks, deps.edn would look like this:

 {:dev {:extra-deps {com.biffweb/build {...}}
        :extra-paths ["dev"]
        :jvm-opts ["-XX:-OmitStackTraceInFastThrow"
        :main-opts ["-m" "build"]}
  :prod {:jvm-opts ["-XX:-OmitStackTraceInFastThrow"
         :main-opts ["-m" "com.example"]}}}

Instead of bb foo, you’d run clj -Mdev foo. All the commands would behave the same as they do now. The default prod setup would have you run clj -Mprod to start the app. Or something. I might fiddle with the specifics. I’m doing compilation automatically, and task startup time is actually looking fine. So I think this’ll work out pretty well–then the only prereq to using Biff is having clj installed; no need to install bb if you haven’t already.

As for config: this part I’m still pondering a fair bit. But at least as a proof-of-concept, I have it set up so you can set a :biff.config/spec key in your system map that describes how config should be merged in from the environment. This is what I have for the example app currently:

(def spec
  {:biff.middleware/cookie-secret {:secret true}
   :biff.xtdb.jdbc/jdbcUrl        {:secret true}
   :biff/jwt-secret               {:secret true}
   :postmark/api-key              {:secret true}
   :recaptcha/secret-key          {:secret true}

   :biff.beholder/enabled  {:default false, :coerce parse-boolean}
   :biff.middleware/secure {:default true, :coerce parse-boolean}
   :biff.xtdb/dir          {:default "storage/xtdb"}
   :biff.xtdb/topology     {:default :standalone, :coerce keyword}
   :biff/base-url          {:default "http://localhost:8080"}
   :biff/host              {:default "localhost", :var-name "HOST"}
   :biff/port              {:default 8080, :coerce parse-long, :var-name "PORT"}
   :postmark/from          {}
   :recaptcha/site-key     {}

   :biff.system-properties/user.timezone                 {:default "UTC"}
   :biff.system-properties/ {:default ""}})


(def initial-system
  {:biff.config/spec config/spec

And just finished writing up a docstring for a use-env-config component which would replace the current use-config and use-secrets components:

  "Merges configuration from the environment (or a file) into the system map.
   Also sets system properties.

   `spec` is a map where each key-value pair corresponds to an environment
   variable. For example:

     {:biff.middleware/cookie-secret {:secret true}
      :biff/port {:default 8080, :coerce parse-long, :var-name \"PORT\"}
      :biff.system-properties/user.timezone {:default \"UTC\"}

   Each key will be added to the system map. The value for each key will be read
   from the environment by converting the key to an environment variable name.
   Periods, slashes, and hyphens are converted to underscores, and letters are
   converted to uppercase. For example:

     :biff.middleware/cookie-secret -> BIFF_MIDDLEWARE_COOKIE_SECRET

   Config value parsing can be modified by setting option keys in the values of
   `spec`. The following options are recognized:

   - :secret
     When true, the config value will converted to a zero-argument function that
     returns the config value (`value` -> `(fn [] value)`). This way, secrets
     won't be exposed if you log the system map.

   - :default
     If the config value is nil or empty, it will be replaced with the value of

   - :coerce
     A one-argument function that will be used to parse the config value, if it
     is set and non-empty.

   - :var-name
     If set, the config value will be read from this environment variable
     instead of inferring an environment variable name as described above.

   Environment variables can also be read from a file by setting the
   biff.env.file system property, for example:

     $ cat env/dev.env

     $ clj -J-Dbiff.env.file=env/dev.env ...

   If a config value is set in both the given file and in the environment, the
   value from the environment will take precedence.

   Any config values that are set to a key with a namespace of
   `biff.system-properties` will be added to the system properties. For example:

     :biff.system-properties/user.timezone {:default \"UTC\"}
     ;; Equivalent to the following, unless you override the default value:
     (System/setProperty \"user.timezone\" \"UTC\")"

When you start a new project, you’ll have env/prod.env and env/dev.env files generated for you, which will look like this (with some of the values filled in):

## env/prod.env
## env/dev.env
# Application

# Tasks

If you want to deploy with, say, an uberjar (there’ll be a clj -Mdev uberjar task by the way), for config you just have to make sure the right env variables are set.

(That is I think the main advantage of going to env variables for config: easier to work with various PaaS things etc. that expect you to pass in config through the environment. )

This might also improve the ergonomics of cloning an existing project and generating config (which I’ve dealt with a fair amount recently in the course of writing how-to articles, which include example repos), though that’s not a fully fleshed-out thought. I guess I like the idea of trying to move more of the “structure” of config into the application source, and then relying on not-checked-in env files only for a few raw values. For some things, setting defaults in the config spec described above can be sufficient; env vars are only needed for cases where you actually need to vary from the defaults.

Also, another small highlight: instead of having you do this to get a secret:

(defn some-handler [{:keys [biff/secret] :as ctx}]
  (let [result (some-api {:api-key (secret :foocorp/api-token)})]

You’d do this:

(defn some-handler [{:keys [foocorp/api-token] :as ctx}]
  (let [result (some-api {:api-key (api-token)})]

i.e. secrets get wrapped in zero-argument functions. A bit more ergonomic, and you’re
still protected from accidentally exposing secrets if you log the system map.

(This, and everything else, would be done in a backwards-compatible way)

Pasting in a message I wrote in Slack, for posterity:

Thanks for the detailed feedback. Currently thinking through this.
At a high level, I think there are basically three situations that I’m thinking about/trying to serve:

  • Private apps deployed with bb deploy (the most common situation for Biff apps I’m sure)
  • Private apps deployed with some other method, e.g. uberjar + docker / some PaaS like
  • Public/open-source apps (including example repos like the ones I’ve written to accompany various how-to guides)

For #1, the current way is fine/probably best. Mainly I’m trying to come up with an approach that will make #2 and #3 smoother. e.g. to combine those two, imagine there’s an open-source Biff app you wanted to deploy to or some other PaaS. I’m trying to figure out what that should look like–what files are checked into the git repo, what are the instructions for people to deploy the app with their own config…

A large part of my thinking was that I wanted to keep as much of the config “structure” as possible checked into the source code, and then have all non-checked-in config be in the form of simple env variables. PaaSes generally have good support for setting config via env vars, so the instructions are just “deploy this repo as is + set your own env vars in the PaaS console/whatever.”

However, this setup is less convenient for #1, which again is the most common situation. Maybe I can structure things a bit differently so you still have a config.edn file like there is now, but it has some reader tags like #env "FOO", #secret "BAR" etc… similar to GitHub - juxt/aero: A small library for explicit, intentful configuration.. And have config.edn be checked into source by default. Then for #1, people can do what they do now–put normal config directly in config.edn and put secrets in secrets.env. But then it’s easy to move any additional config values out of config.edn and into environment variables if you want.

In fact maybe just using Aero to parse config.edn would do the trick. Would be worth thinking about/experimenting with. I am wondering if it has too many features though and defining a couple of simple #env and #secret reader tags would be all Biff needs.

Anyway I’ll do a second take on a proof-of-concept and report back!

I’ve wrote up an example of what using Aero would look like, and I think it’s really nice. config.edn would be moved to resources/config.edn (so it’ll get bundled into a jar if you deploy your app that way) and will be checked into source. For new Biff apps it’ll start out looking like this:

;; See
{:biff/base-url #profile {:prod #join ["https://" #biff/env DOMAIN]
                          :dev "http://localhost:8080"}
 :biff/host     #profile {:prod "localhost" :dev ""}
 :biff/port     8080

 :biff.xtdb/dir           "storage/xtdb"
 :biff.xtdb/topology      :standalone
 ;; Standalone topology in production isn't recommended for anything
 ;; serious. You can uncomment the following to use managed postgres
 ;; instead.
 ;; :biff.xtdb/topology      #profile {:prod :jdbc :dev :standalone}
 ;; :biff.xtdb.jdbc/user     "user"
 ;; :biff.xtdb.jdbc/password #biff/secret JDBC_PASSWORD ; Set this in config.env
 ;; :biff.xtdb.jdbc/host     "host"
 ;; :biff.xtdb.jdbc/port     5432
 ;; :biff.xtdb.jdbc/dbname   "dbname"

 :biff.beholder/enabled         #profile {:prod false :dev true}
 :biff.middleware/cookie-secret #biff/secret COOKIE_SECRET
 :biff.middleware/secure        #profile {:prod true :dev false}
 :biff/jwt-secret               #biff/secret JWT_SECRET

 ;; Postmark is used to send email sign-in links. Create an account at
 :postmark/api-key #biff/secret POSTMARK_API_KEY
 ;; Change to the address of your sending identity. Set a reply-to
 ;; address on your sending identity if you want to receive replies and
 ;; your from address isn't configured for receiving.
 :postmark/from    #biff/env POSTMARK_FROM

 ;; Recaptcha is used to protect your sign-in page. Go to
 ;; and add a site. Select v2
 ;; invisible. Add localhost to your list of allowed domains.
 :recaptcha/secret-key #biff/secret RECAPTCHA_SECRET_KEY
 :recaptcha/site-key   #biff/env RECAPTCHA_SITE_KEY

 :biff.system-properties/user.timezone                 "UTC"
 :biff.system-properties/ ""

 :biff.tasks/server         #biff/env DOMAIN
 :biff.tasks/main-ns        com.example
 :biff.tasks/on-soft-deploy "\"(com.example/on-save @com.example/system)\""

 ;; Set this if the auto-detection doesn't work. See
 ;; for possible values (for
 ;; example: "tailwindcss-macos-arm64").
 :biff.tasks/tailwind-file nil

 ;; Uncomment this line if you're on Windows/don't have rsync and your local branch is
 ;; called main instead of master:
 ;; :biff.tasks/deploy-cmd ["git" "push" "prod" "main:master"]
 :biff.tasks/deploy-cmd ["git" "push" "prod" "master"]
 ;; Uncomment this line if you have any ssh-related problems:
 ;; :biff.tasks/skip-ssh-agent true

Then you’ll have a config.env file at the root of your project which contains any environment variables, and isn’t checked into source. it’s the same as the current secrets.env, just renamed since you can use it for normal config too, not just secrets.

Instead of separating the config into separate sections for each environment (:prod, :dev, …), we use Aero’s #profile reader tag. I was worried it might be more verbose, but actually it’s not bad. It’s more explicit too–you don’t have to look at multiple places in the config file to know what the value of a particular config key will be. The profile will be set to (keyword (System/getProperty "biff.profile")). It no longer defaults to :prod–you’ll need to do clj -J-Dbiff.profile=prod ... / java -Dbiff.profile=prod ... explicitly (handled for you by bb dev / / the upcoming Dockerfile).

Note the #biff/env and #biff/secret usages. #biff/env is like Aero’s #env, except it also supports reading env variables from a config.env file if it exists. #biff/secret is like #biff/env except that it wraps values in a zero arg function (only if defined though–if the key isn’t set, #biff/secret will return nil, so as to play nicely with #profile and #or).

So you could do e.g. (let [api-key ((:postmark/api-key ctx))] ...) to get a secret. The :biff/secret function will still be set, so you can also do:

(defn my-handler [{:keys [biff/secret]}]
  (let [api-key (secret :postmark/api-key)]

That’s a better approach if the secret might be nil: in that case, (secret :postmark/api-key) would return nil, whereas ((:postmark/api-key ctx)) would throw an exception.

Previously, the :biff/secret fn took ctx as its first argument (e.g. (secret ctx :postmark/api-key)). Now I’ve made that function close over ctx, so you only have to pass in the keyword. For backwards-compatibility, you can still pass in ctx, but it’ll be ignored.

Finally, by default system properties are set at runtime instead of being passed in as CLI arguments. Just add them to config.edn and use the biff.system-properties namespace (e.g. :biff.system-properties/user.timezone "UTC").

Implementation for anyone interested:

(defmethod aero/reader 'biff/env
  [{:keys [profile] :as opts} _ value]
  (not-empty (get env (str value))))

(defmethod aero/reader 'biff/secret
  [{:keys [profile] :as opts} _ value]
  (when-some [value (aero/reader opts 'biff/env value)]
    (fn [] value)))

(defn use-aero-config [ctx]
  (let [profile (keyword (System/getProperty "biff.profile"))
        env (merge (some->> (util/catchall (slurp "config.env"))
                            (keep parse-env-var)
                            (into {}))
                   (into {} (System/getenv)))
        ctx (merge ctx (aero/read-config (io/resource "config.edn") {:profile profile env}))
        secret-fn (fn get-secret
                    ([k] (some-> (get ctx k) (.invoke)))
                    ;; Backwards compatibility
                    ([ctx k] (get-secret k)))
        ctx (assoc ctx :biff/secret secret-fn)]
    (when-not (and (secret-fn :biff.middleware/cookie-secret)
                   (secret-fn :biff/jwt-secret))
      (binding [*out* *err*]
        (println "Secrets are missing. You may need to run `clj -Mdev generate-secrets` "
                 "and then either edit config.env or set them via environment variables.")
        (System/exit 1)))
    (doseq [[k v] (util-ns/select-ns-as ctx 'biff.system-properties nil)]
      (System/setProperty (name k) v))

Taking a step back: this gives you the ability to decide on a per-project basis what config to put into env variables and what config to check into source. I’ve done a mix in the template project’s resources/config.edn file above, but anywhere you have a #biff/env, it’s fine if you want to take the value from config.env and stick it directly into resources/config.edn.