[feature] Allow loading TLS certs from disk (#1586)

Currently, GtS only supports using the built-in LE client directly for
TLS. However, admins may still want to use GtS directly (so without a
reverse proxy) but with certificates provided through some other
mechanism. They may have some centralised way of provisioning these
things themselves, or simply prefer to use LE but with a different
challenge like DNS-01 which is not supported by autocert.

This adds support for loading a public/private keypair from disk instead
of using LE and reconfigures the server to use a TLS listener if we
succeed in doing so.

Additionally, being able to load TLS keypair from disk opens up the path
to using a custom CA for testing purposes avoinding the need for a
constellation of containers and something like Pebble or Step CA to
provide LE APIs.

CONTRIBUTING.md

@@ -29,6 +29,7 @@ These contribution guidelines were adapted from / inspired by those of Gitea (ht
       - [SQLite](#sqlite)
       - [Postgres](#postgres)
     - [CLI Tests](#cli-tests)
+    - [Federation](#federation)
   - [Updating Swagger docs](#updating-swagger-docs)
   - [CI/CD configuration](#cicd-configuration)
   - [Release Checklist](#release-checklist)
@@ -418,6 +419,20 @@ In [./test/envparsing.sh](./test/envparsing.sh) there's a test for making sure t
 
 Although this test *is* part of the CI/CD testing process, you probably won't need to worry too much about running it yourself. That is, unless you're messing about with code inside the `main` package in `cmd/gotosocial`, or inside the `config` package in `internal/config`.
 
+#### Federation
+
+By using the support for loading TLS files from disk it is possible to have two local instances with TLS to allow for (manually) testing federation.
+
+You'll need to set the following configuration options:
+* `GTS_TLS_CERTIFICATE_CHAIN`: poiting to a PEM-encoded certificate chain including the public certificate
+* `GTS_TLS_CERTIFICATE_KEY`: pointing to a PEM-encoded private key
+
+Additionally, for the Go HTTP client to recognise certificates issued by a custom CA as valid, you'll need to set one of:
+* `SSL_CERT_FILE`: pointing to the public key of your custom CA
+* `SSL_CERT_DIR`: a `:`-separated list of directories to load CA certificates from
+
+You'll additionally need functioning DNS for your two instance names which you can achieve through entries in `/etc/hosts` or by running a local DNS server like [dnsmasq](https://thekelleys.org.uk/dnsmasq/doc.html).
+
 ### Updating Swagger docs
 
 GoToSocial uses [go-swagger](https://goswagger.io) to generate Swagger API documentation from code annotations.

docs/configuration/tls.md

@@ -1,4 +1,12 @@
-# LetsEncrypt
+# TLS
+
+It's possible to configure TLS support in one of two ways:
+* Built-in support for Lets Encrypt / ACME compatible vendors
+* Loading TLS files from disk
+
+It is not possible to have both methods enabled at the same time.
+
+Note that when using TLS files loaded from disk you are responsible for restarting the instance when the files change. They are not automatically reloaded.
 
 ## Settings
 
@@ -39,4 +47,20 @@ letsencrypt-cert-dir: "/gotosocial/storage/certs"
 # Examples: ["admin@example.org"]
 # Default: ""
 letsencrypt-email-address: ""
+
+##############################
+##### MANUAL TLS CONFIG  #####
+##############################
+
+# String. Path to a PEM-encoded file on disk that includes the certificate chain 
+# and the public key
+# Examples: ["/gotosocial/storage/certs/chain.pem"]
+# Default: ""
+tls-certificate-chain: ""
+
+# String. Path to a PEM-encoded file on disk containing the private key for the
+# associated tls-certificate-chain
+# Examples: ["/gotosocial/storage/certs/private.pem"]
+# Default: ""
+tls-certificate-key: ""
 ```

docs/installation_guide/docker.md

@@ -92,7 +92,7 @@ For example, let's say you created the `~/gotosocial/data` directory for a user
 
 #### LetsEncrypt (optional)
 
-If you want to use [LetsEncrypt](../configuration/letsencrypt.md) for ssl certificates (https), you should also:
+If you want to use [LetsEncrypt](../configuration/tls.md) for ssl certificates (https), you should also:
 
 1. Change the value of `GTS_LETSENCRYPT_ENABLED` to `"true"`.
 2. Remove the `#` before `- "80:80"` in the `ports` section.

example/config.yaml

@@ -575,6 +575,22 @@ letsencrypt-cert-dir: "/gotosocial/storage/certs"
 # Default: ""
 letsencrypt-email-address: ""
 
+##############################
+##### MANUAL TLS CONFIG  #####
+##############################
+
+# String. Path to a PEM-encoded file on disk that includes the certificate chain 
+# and the public key
+# Examples: ["/gotosocial/storage/certs/chain.pem"]
+# Default: ""
+tls-certificate-chain: ""
+
+# String. Path to a PEM-encoded file on disk containing the private key for the
+# associated tls-certificate-chain
+# Examples: ["/gotosocial/storage/certs/private.pem"]
+# Default: ""
+tls-certificate-key: ""
+
 #######################
 ##### OIDC CONFIG #####
 #######################

internal/config/config.go

@@ -114,6 +114,9 @@ type Configuration struct {
 	LetsEncryptCertDir      string `name:"letsencrypt-cert-dir" usage:"Directory to store acquired letsencrypt certificates."`
 	LetsEncryptEmailAddress string `name:"letsencrypt-email-address" usage:"Email address to use when requesting letsencrypt certs. Will receive updates on cert expiry etc."`
 
+	TLSCertificateChain string `name:"tls-certificate-chain" usage:"Filesystem path to the certificate chain including any intermediate CAs and the TLS public key"`
+	TLSCertificateKey   string `name:"tls-certificate-key" usage:"Filesystem path to the TLS private key"`
+
 	OIDCEnabled          bool     `name:"oidc-enabled" usage:"Enabled OIDC authorization for this instance. If set to true, then the other OIDC flags must also be set."`
 	OIDCIdpName          string   `name:"oidc-idp-name" usage:"Name of the OIDC identity provider. Will be shown to the user when logging in."`
 	OIDCSkipVerification bool     `name:"oidc-skip-verification" usage:"Skip verification of tokens returned by the OIDC provider. Should only be set to 'true' for testing purposes, never in a production environment!"`

internal/config/defaults.go

@@ -91,6 +91,9 @@ var Defaults = Configuration{
 	LetsEncryptCertDir:      "/gotosocial/storage/certs",
 	LetsEncryptEmailAddress: "",
 
+	TLSCertificateChain: "",
+	TLSCertificateKey:   "",
+
 	OIDCEnabled:          false,
 	OIDCIdpName:          "",
 	OIDCSkipVerification: false,

internal/config/flags.go

@@ -114,6 +114,10 @@ func (s *ConfigState) AddServerFlags(cmd *cobra.Command) {
 		cmd.Flags().String(LetsEncryptCertDirFlag(), cfg.LetsEncryptCertDir, fieldtag("LetsEncryptCertDir", "usage"))
 		cmd.Flags().String(LetsEncryptEmailAddressFlag(), cfg.LetsEncryptEmailAddress, fieldtag("LetsEncryptEmailAddress", "usage"))
 
+		// Manual TLS
+		cmd.Flags().String(TLSCertificateChainFlag(), cfg.TLSCertificateChain, fieldtag("TLSCertificateChain", "usage"))
+		cmd.Flags().String(TLSCertificateKeyFlag(), cfg.TLSCertificateKey, fieldtag("TLSCertificateKey", "usage"))
+
 		// OIDC
 		cmd.Flags().Bool(OIDCEnabledFlag(), cfg.OIDCEnabled, fieldtag("OIDCEnabled", "usage"))
 		cmd.Flags().String(OIDCIdpNameFlag(), cfg.OIDCIdpName, fieldtag("OIDCIdpName", "usage"))

internal/config/helpers.gen.go

@@ -1524,6 +1524,56 @@ func GetLetsEncryptEmailAddress() string { return global.GetLetsEncryptEmailAddr
 // SetLetsEncryptEmailAddress safely sets the value for global configuration 'LetsEncryptEmailAddress' field
 func SetLetsEncryptEmailAddress(v string) { global.SetLetsEncryptEmailAddress(v) }
 
+// GetTLSCertificateChain safely fetches the Configuration value for state's 'TLSCertificateChain' field
+func (st *ConfigState) GetTLSCertificateChain() (v string) {
+	st.mutex.Lock()
+	v = st.config.TLSCertificateChain
+	st.mutex.Unlock()
+	return
+}
+
+// SetTLSCertificateChain safely sets the Configuration value for state's 'TLSCertificateChain' field
+func (st *ConfigState) SetTLSCertificateChain(v string) {
+	st.mutex.Lock()
+	defer st.mutex.Unlock()
+	st.config.TLSCertificateChain = v
+	st.reloadToViper()
+}
+
+// TLSCertificateChainFlag returns the flag name for the 'TLSCertificateChain' field
+func TLSCertificateChainFlag() string { return "tls-certificate-chain" }
+
+// GetTLSCertificateChain safely fetches the value for global configuration 'TLSCertificateChain' field
+func GetTLSCertificateChain() string { return global.GetTLSCertificateChain() }
+
+// SetTLSCertificateChain safely sets the value for global configuration 'TLSCertificateChain' field
+func SetTLSCertificateChain(v string) { global.SetTLSCertificateChain(v) }
+
+// GetTLSCertificateKey safely fetches the Configuration value for state's 'TLSCertificateKey' field
+func (st *ConfigState) GetTLSCertificateKey() (v string) {
+	st.mutex.Lock()
+	v = st.config.TLSCertificateKey
+	st.mutex.Unlock()
+	return
+}
+
+// SetTLSCertificateKey safely sets the Configuration value for state's 'TLSCertificateKey' field
+func (st *ConfigState) SetTLSCertificateKey(v string) {
+	st.mutex.Lock()
+	defer st.mutex.Unlock()
+	st.config.TLSCertificateKey = v
+	st.reloadToViper()
+}
+
+// TLSCertificateKeyFlag returns the flag name for the 'TLSCertificateKey' field
+func TLSCertificateKeyFlag() string { return "tls-certificate-key" }
+
+// GetTLSCertificateKey safely fetches the value for global configuration 'TLSCertificateKey' field
+func GetTLSCertificateKey() string { return global.GetTLSCertificateKey() }
+
+// SetTLSCertificateKey safely sets the value for global configuration 'TLSCertificateKey' field
+func SetTLSCertificateKey(v string) { global.SetTLSCertificateKey(v) }
+
 // GetOIDCEnabled safely fetches the Configuration value for state's 'OIDCEnabled' field
 func (st *ConfigState) GetOIDCEnabled() (v bool) {
 	st.mutex.Lock()

internal/config/validate.go

@@ -67,6 +67,19 @@ func Validate() error {
 		errs = append(errs, fmt.Errorf("%s must be set", WebAssetBaseDirFlag()))
 	}
 
+	tlsChain := GetTLSCertificateChain()
+	tlsKey := GetTLSCertificateKey()
+	tlsChainFlag := TLSCertificateChainFlag()
+	tlsKeyFlag := TLSCertificateKeyFlag()
+
+	if GetLetsEncryptEnabled() && (tlsChain != "" || tlsKey != "") {
+		errs = append(errs, fmt.Errorf("%s cannot be enabled when %s and/or %s are also set", LetsEncryptEnabledFlag(), tlsChainFlag, tlsKeyFlag))
+	}
+
+	if (tlsChain != "" && tlsKey == "") || (tlsChain == "" && tlsKey != "") {
+		errs = append(errs, fmt.Errorf("%s and %s need to both be set or unset", tlsChainFlag, tlsKeyFlag))
+	}
+
 	if len(errs) > 0 {
 		errStrings := []string{}
 		for _, err := range errs {

internal/router/router.go

@@ -20,6 +20,7 @@ package router
 
 import (
 	"context"
+	"crypto/tls"
 	"fmt"
 	"net"
 	"net/http"
@@ -78,6 +79,26 @@ func (r *router) Start() {
 	// but updated to TLS if LetsEncrypt is enabled.
 	listen := r.srv.ListenAndServe
 
+	// During config validation we already checked that both Chain and Key are set
+	// so we can forego checking for both here
+	if chain := config.GetTLSCertificateChain(); chain != "" {
+		pkey := config.GetTLSCertificateKey()
+		cer, err := tls.LoadX509KeyPair(chain, pkey)
+		if err != nil {
+			log.Fatalf(
+				nil,
+				"tls: failed to load keypair from %s and %s, ensure they are PEM-encoded and can be read by this process: %s",
+				chain, pkey, err,
+			)
+		}
+		r.srv.TLSConfig = &tls.Config{
+			MinVersion:   tls.VersionTLS12,
+			Certificates: []tls.Certificate{cer},
+		}
+		// TLS is enabled, update the listen function
+		listen = func() error { return r.srv.ListenAndServeTLS("", "") }
+	}
+
 	if config.GetLetsEncryptEnabled() {
 		// LetsEncrypt support is enabled
 

mkdocs.yml

@@ -65,7 +65,7 @@ nav:
       - "configuration/media.md"
       - "configuration/storage.md"
       - "configuration/statuses.md"
-      - "configuration/letsencrypt.md"
+      - "configuration/tls.md"
       - "configuration/oidc.md"
       - "configuration/smtp.md"
       - "configuration/syslog.md"

test/envparsing.sh

@@ -2,7 +2,7 @@
 
 set -eu
 
-EXPECT='{"account-domain":"peepee","accounts-allow-custom-css":true,"accounts-approval-required":false,"accounts-reason-required":false,"accounts-registration-open":true,"advanced-cookies-samesite":"strict","advanced-rate-limit-requests":6969,"advanced-throttling-multiplier":-1,"advanced-throttling-retry-after":10000000000,"application-name":"gts","bind-address":"127.0.0.1","cache":{"gts":{"account-max-size":99,"account-sweep-freq":1000000000,"account-ttl":10800000000000,"block-max-size":100,"block-sweep-freq":30000000000,"block-ttl":300000000000,"domain-block-max-size":1000,"domain-block-sweep-freq":60000000000,"domain-block-ttl":86400000000000,"emoji-category-max-size":100,"emoji-category-sweep-freq":30000000000,"emoji-category-ttl":300000000000,"emoji-max-size":500,"emoji-sweep-freq":30000000000,"emoji-ttl":300000000000,"media-max-size":500,"media-sweep-freq":30000000000,"media-ttl":300000000000,"mention-max-size":500,"mention-sweep-freq":30000000000,"mention-ttl":300000000000,"notification-max-size":500,"notification-sweep-freq":30000000000,"notification-ttl":300000000000,"report-max-size":100,"report-sweep-freq":30000000000,"report-ttl":300000000000,"status-max-size":500,"status-sweep-freq":30000000000,"status-ttl":300000000000,"tombstone-max-size":100,"tombstone-sweep-freq":30000000000,"tombstone-ttl":300000000000,"user-max-size":100,"user-sweep-freq":30000000000,"user-ttl":300000000000}},"config-path":"internal/config/testdata/test.yaml","db-address":":memory:","db-database":"gotosocial_prod","db-max-open-conns-multiplier":3,"db-password":"hunter2","db-port":6969,"db-sqlite-busy-timeout":1000000000,"db-sqlite-cache-size":0,"db-sqlite-journal-mode":"DELETE","db-sqlite-synchronous":"FULL","db-tls-ca-cert":"","db-tls-mode":"disable","db-type":"sqlite","db-user":"sex-haver","dry-run":true,"email":"","host":"example.com","instance-deliver-to-shared-inboxes":false,"instance-expose-peers":true,"instance-expose-public-timeline":true,"instance-expose-suspended":true,"instance-expose-suspended-web":true,"landing-page-user":"admin","letsencrypt-cert-dir":"/gotosocial/storage/certs","letsencrypt-email-address":"","letsencrypt-enabled":true,"letsencrypt-port":80,"log-db-queries":true,"log-level":"info","media-description-max-chars":5000,"media-description-min-chars":69,"media-emoji-local-max-size":420,"media-emoji-remote-max-size":420,"media-image-max-size":420,"media-remote-cache-days":30,"media-video-max-size":420,"oidc-admin-groups":["steamy"],"oidc-client-id":"1234","oidc-client-secret":"shhhh its a secret","oidc-enabled":true,"oidc-idp-name":"sex-haver","oidc-issuer":"whoknows","oidc-link-existing":true,"oidc-scopes":["read","write"],"oidc-skip-verification":true,"password":"","path":"","port":6969,"protocol":"http","request-id-header":"X-Trace-Id","smtp-from":"queen.rip.in.piss@terfisland.org","smtp-host":"example.com","smtp-password":"hunter2","smtp-port":4269,"smtp-username":"sex-haver","software-version":"","statuses-cw-max-chars":420,"statuses-max-chars":69,"statuses-media-max-files":1,"statuses-poll-max-options":1,"statuses-poll-option-max-chars":50,"storage-backend":"local","storage-local-base-path":"/root/store","storage-s3-access-key":"minio","storage-s3-bucket":"gts","storage-s3-endpoint":"localhost:9000","storage-s3-proxy":true,"storage-s3-secret-key":"miniostorage","storage-s3-use-ssl":false,"syslog-address":"127.0.0.1:6969","syslog-enabled":true,"syslog-protocol":"udp","trusted-proxies":["127.0.0.1/32","docker.host.local"],"username":"","web-asset-base-dir":"/root","web-template-base-dir":"/root"}'
+EXPECT='{"account-domain":"peepee","accounts-allow-custom-css":true,"accounts-approval-required":false,"accounts-reason-required":false,"accounts-registration-open":true,"advanced-cookies-samesite":"strict","advanced-rate-limit-requests":6969,"advanced-throttling-multiplier":-1,"advanced-throttling-retry-after":10000000000,"application-name":"gts","bind-address":"127.0.0.1","cache":{"gts":{"account-max-size":99,"account-sweep-freq":1000000000,"account-ttl":10800000000000,"block-max-size":100,"block-sweep-freq":30000000000,"block-ttl":300000000000,"domain-block-max-size":1000,"domain-block-sweep-freq":60000000000,"domain-block-ttl":86400000000000,"emoji-category-max-size":100,"emoji-category-sweep-freq":30000000000,"emoji-category-ttl":300000000000,"emoji-max-size":500,"emoji-sweep-freq":30000000000,"emoji-ttl":300000000000,"media-max-size":500,"media-sweep-freq":30000000000,"media-ttl":300000000000,"mention-max-size":500,"mention-sweep-freq":30000000000,"mention-ttl":300000000000,"notification-max-size":500,"notification-sweep-freq":30000000000,"notification-ttl":300000000000,"report-max-size":100,"report-sweep-freq":30000000000,"report-ttl":300000000000,"status-max-size":500,"status-sweep-freq":30000000000,"status-ttl":300000000000,"tombstone-max-size":100,"tombstone-sweep-freq":30000000000,"tombstone-ttl":300000000000,"user-max-size":100,"user-sweep-freq":30000000000,"user-ttl":300000000000}},"config-path":"internal/config/testdata/test.yaml","db-address":":memory:","db-database":"gotosocial_prod","db-max-open-conns-multiplier":3,"db-password":"hunter2","db-port":6969,"db-sqlite-busy-timeout":1000000000,"db-sqlite-cache-size":0,"db-sqlite-journal-mode":"DELETE","db-sqlite-synchronous":"FULL","db-tls-ca-cert":"","db-tls-mode":"disable","db-type":"sqlite","db-user":"sex-haver","dry-run":true,"email":"","host":"example.com","instance-deliver-to-shared-inboxes":false,"instance-expose-peers":true,"instance-expose-public-timeline":true,"instance-expose-suspended":true,"instance-expose-suspended-web":true,"landing-page-user":"admin","letsencrypt-cert-dir":"/gotosocial/storage/certs","letsencrypt-email-address":"","letsencrypt-enabled":true,"letsencrypt-port":80,"log-db-queries":true,"log-level":"info","media-description-max-chars":5000,"media-description-min-chars":69,"media-emoji-local-max-size":420,"media-emoji-remote-max-size":420,"media-image-max-size":420,"media-remote-cache-days":30,"media-video-max-size":420,"oidc-admin-groups":["steamy"],"oidc-client-id":"1234","oidc-client-secret":"shhhh its a secret","oidc-enabled":true,"oidc-idp-name":"sex-haver","oidc-issuer":"whoknows","oidc-link-existing":true,"oidc-scopes":["read","write"],"oidc-skip-verification":true,"password":"","path":"","port":6969,"protocol":"http","request-id-header":"X-Trace-Id","smtp-from":"queen.rip.in.piss@terfisland.org","smtp-host":"example.com","smtp-password":"hunter2","smtp-port":4269,"smtp-username":"sex-haver","software-version":"","statuses-cw-max-chars":420,"statuses-max-chars":69,"statuses-media-max-files":1,"statuses-poll-max-options":1,"statuses-poll-option-max-chars":50,"storage-backend":"local","storage-local-base-path":"/root/store","storage-s3-access-key":"minio","storage-s3-bucket":"gts","storage-s3-endpoint":"localhost:9000","storage-s3-proxy":true,"storage-s3-secret-key":"miniostorage","storage-s3-use-ssl":false,"syslog-address":"127.0.0.1:6969","syslog-enabled":true,"syslog-protocol":"udp","tls-certificate-chain":"","tls-certificate-key":"","trusted-proxies":["127.0.0.1/32","docker.host.local"],"username":"","web-asset-base-dir":"/root","web-template-base-dir":"/root"}'
 
 # Set all the environment variables to 
 # ensure that these are parsed without panic