Simple Hack on Kubernetes
One container image, one pod, your Postgres, your load balancer. This is the
runbook for an internal hackathon run inside a company: the chart is
deploy/kubernetes/simple-hack/, the same chart rendered to plain YAML is
deploy/kubernetes/manifests/simple-hack.yaml, and the image is
ghcr.io/vineetu/simple-host:hack-<release> (the Simple Hack release build;
0.8.14 is the first that can build its own database, which this needs).
The pod runs simple-host in Hack mode (EVENTS=hosted): organisers sign in
at https://<host>/ and create events, each event’s page is
https://<event>.<host>/, and each team’s site is its own origin,
https://<team>.<event>.<host>/ (so no team’s page can read another’s
visitor sessions). TLS ends at your load balancer; the pod speaks plain HTTP
on 8090.
Running many events at once and hitting your load balancer’s or
certificate’s per-name limits (prerequisite 3 below)? Set
teamSites.flatHosts: true and a team’s site becomes
https://<team>--<event>.<host>/ instead — one DNS label, so the same
*.<host> wildcard DNS, certificate and Ingress rule you already have for
event pages covers every team site too, with no per-event certificate SAN
or helm upgrade. Off by default, matching simple-hack.app.
Prerequisites
Have these before helm install; the chart cannot make them.
-
A Postgres 16 database and a role that owns it, not a superuser. No extensions.
CREATE ROLE hack LOGIN PASSWORD '...'; CREATE DATABASE hack OWNER hack;The pod builds the schema in an empty database on its first start and applies migrations on every start (
MIGRATE_ON_START=true, under a Postgres advisory lock, so a second pod starting at the same moment waits). Keep the connection string handy:postgres://hack:PASSWORD@<pg-host>:5432/hack?sslmode=require. The server opens at most 25 connections (DB_MAX_OPEN_CONNS). -
DNS: the apex and one wildcard, both to the address of the load balancer in front of your ingress controller:
hack.example.comand*.hack.example.com. A DNS wildcard matches more than one label, so that one record also resolvesteam.event.hack.example.com. Inside a company these are records in the internal zone. Checkhack.example.com,anything.hack.example.comanda.b.hack.example.comall resolve before installing. -
The load balancer and TLS. A certificate for
hack.example.com,*.hack.example.com(event pages) and, for each event you will run,*.<event>.hack.example.com(its team sites): a wildcard certificate covers one label only, so the event’s wildcard has to be a SAN of its own. Decide the event’s slug (what the organiser types when creating it;spring-hackgivesspring-hack.hack.example.com) and get the certificate issued with that SAN. Inside a company that is one request to the internal CA; a public CA issues it too (Let’s Encrypt with a DNS challenge). Terminate TLS at the load balancer or the ingress controller. Whatever terminates it must forwardX-Forwarded-Proto: httpsand append the client address toX-Forwarded-For; every common load balancer and ingress controller does. WithoutX-Forwarded-Protothe sign-in cookies lose theirSecureflag; without a saneX-Forwarded-Forevery participant shares one rate limit. Uploads are up to 64 MB per request (MAX_ARCHIVE_MB, default 25 in Hack, plus overhead): raise the proxy’s body limit to match (the chart’s ingress-nginx annotations do).The Ingress needs the same list: an Ingress host wildcard also matches one label, so the chart adds a
*.<event>.hack.example.comrule for every slug underingress.events. An event created later is one line and ahelm upgradeaway (oringress.catchAll: truewhen the controller is this app’s alone).teamSites.flatHosts: trueskips all of that. A team’s site becomes the single label<team>--<event>.hack.example.cominstead of<team>.<event>.hack.example.com, which thehack.example.comand*.hack.example.comcertificate and Ingress rule from the start of this step already cover — no per-event SAN, noingress.eventsentry, nohelm upgradeper event. Running against AWS ALB specifically: this is the escape hatch for its per-certificate SAN and per-listener rule limits, which a growing roster of concurrent events runs into otherwise. Turn it on for a new install, or before any event exists: existing two-level team addresses are not redirected to the flat form, so turning it on after events are already running leaves their team sites at the old addresses. -
An OIDC client at your identity provider (Entra ID, Okta, Google Workspace, Keycloak): a confidential web client, authorization-code grant, with exactly this redirect URI:
https://hack.example.com/v1/auth/oauth/oidc/callbackNote the issuer URL, client ID and client secret. Provider-by-provider steps are in simple-hack-standalone.md. Without an identity provider, password sign-in works by default; optionally use emailed codes through Resend (
signIn.email), which needs the pod to reach api.resend.com. -
A storage class that can give a ReadWriteOnce volume (every default one can). The volume holds every site; 20 GB is plenty for a hackathon.
-
kubectlandhelm3 pointed at the cluster, and a namespace.
Install
Copy deploy/kubernetes/examples/values-example.yaml
to my-values.yaml and fill in the “set this” lines (host,
database.dsn, signIn.oidc.issuer, clientId, clientSecret) and the
event slug under ingress.events. Every other value is documented in
values.yaml. Then:
helm install simple-hack deploy/kubernetes/simple-hack \
--namespace simple-hack --create-namespace \
-f my-values.yaml
kubectl -n simple-hack rollout status deploy/simple-hack
kubectl -n simple-hack logs deploy/simple-hack
The log shows MIGRATE_ON_START: building the schema if the database is empty, built the schema from host/db/schema.sql, the migrations being
recorded, then hosted events: on (the hackathon platform at https://hack.example.com) and the listen line. The pod is Ready when
/readyz can ping Postgres.
Check from outside:
curl -sS https://hack.example.com/healthz # {"status":"ok"}
curl -sS -o /dev/null -w '%{http_code}\n' https://hack.example.com/ # 200
Without Helm: create the Secret from
manifests/secret.example.yaml,
edit the hostnames in
manifests/simple-hack.yaml,
and kubectl apply -f both. Re-render your own copy with
deploy/kubernetes/render.sh after changing the example values.
Pod security
The pod meets the Kubernetes restricted Pod Security Standard: non-root
(uid 65532), read-only root filesystem, no capabilities, seccomp
RuntimeDefault, no service-account token. The namespace can carry
pod-security.kubernetes.io/enforce=restricted. It writes only to /data
(the volume) and /tmp (an emptyDir).
First sign-in and the first admin
There is no admin step. The first person to open https://hack.example.com/
signs in with the company account, lands on their home page and creates the
event; they are its organiser. Anyone else who signs in and has the event’s
join link, judge link or applies from the event page takes part. Nothing
about the first account is special.
The platform admin is not a person: it is the ADMIN_API_KEY, used as an
X-API-Key header against /v1/admin/* (list every event, take a site
down, create key-only accounts for a bot or a test). The chart generated it
on the first install and keeps it across upgrades:
kubectl -n simple-hack get secret simple-hack-secrets \
-o jsonpath='{.data.ADMIN_API_KEY}' | base64 -d
Keep it out of chat and tickets. With signIn.email.enabled: "off" (the
example values), password accounts still work by default. For company accounts
only, also set signIn.password.enabled: false.
signIn.oidc.allowedEmailDomains refuses anyone outside the listed domains
even when the provider would have let them through.
Upgrades
A release is a new image tag. Migrations run on start, so an upgrade is:
helm upgrade simple-hack deploy/kubernetes/simple-hack \
--namespace simple-hack -f my-values.yaml \
--set image.tag=hack-0.8.15
kubectl -n simple-hack rollout status deploy/simple-hack
The Deployment’s strategy is Recreate: the old pod stops, the new one
starts, applies the release’s pending migrations, then listens. Expect about
thirty seconds of downtime per upgrade; do it between sessions, not during
judging. A migration that fails leaves the pod in CrashLoopBackOff with
the reason in its log and the database as it was (each file runs in its own
transaction); helm rollback simple-hack brings the previous image back,
and a database the newer release already migrated still runs the older
binary (migrations are additive).
Changing a value restarts the pod (the settings are read at start). The
generated secrets (ADMIN_API_KEY, PASSCODE_ENC_KEY, ANALYTICS_SALT)
are read back from the live Secret on every upgrade and never rotate on
their own; the Secret and the volume claim survive helm uninstall.
To prepare a database by hand instead (migrateOnStart: false), run the
same image once:
kubectl -n simple-hack run migrate --rm -it --restart=Never \
--image ghcr.io/vineetu/simple-host:hack-0.8.14 \
--env DB_DSN="$(kubectl -n simple-hack get secret simple-hack-secrets -o jsonpath='{.data.DB_DSN}' | base64 -d)" \
-- migrate -init
Backups
Two things hold state, and a restore needs both from about the same time:
-
Postgres: accounts, events, teams, scores, which site version is live. Your database’s own backups cover it; by hand,
pg_dump -Fc "$DB_DSN" > hack.dumpandpg_restore -d "$DB_DSN" hack.dump. -
The volume (
/data): every site’s files and versions, and each site’s SQLite, KV and uploaded files. Take a VolumeSnapshot of the claimsimple-hack-data(your CSI driver’s class), or copy it out:kubectl -n simple-hack exec deploy/simple-hack -- tar -C /data -cf - . > data.tar(
taris not in the image; usekubectl cpor a debug container with the volume mounted, or the snapshot.)
Take the two together while the pod is scaled to zero
(kubectl -n simple-hack scale deploy/simple-hack --replicas=0) for a
consistent pair; during an event, a live pg_dump plus a snapshot a minute
apart is fine (a site saved in between is missing from one and the server
tolerates it). To restore: scale to zero, pg_restore, restore the snapshot
into a new claim, set persistence.existingClaim, scale back up.
A copy of the Secret belongs with the backups: PASSCODE_ENC_KEY seals every
site passcode and notification secret, and the data is unreadable without it.
What doesn’t work without what
| Without | What happens |
|---|---|
| The wildcard DNS record | https://hack.example.com/ works; every event page and team site is unreachable. The platform’s home page still hands out those addresses. |
A certificate that covers *.hack.example.com |
Event pages warn or fail in the browser. |
The event’s *.<event>.hack.example.com SAN in that certificate (teamSites.flatHosts: false, the default) |
Team sites warn or fail in the browser; the rest of the event (sign-up, teams, judging) works. |
The event’s slug under ingress.events (or ingress.catchAll), with teamSites.flatHosts: false |
The ingress controller answers https://<team>.<event>.hack.example.com/ with its default backend (404). Add the slug and helm upgrade, or turn teamSites.flatHosts on instead. |
Team sites at <event>.hack.example.com/<team>/ |
Not served, by design: each team is its own origin so no team’s page can read another’s visitor sessions or storage. |
X-Forwarded-Proto: https from the proxy |
Sign-in works; cookies are set without Secure. |
A trustworthy X-Forwarded-For (or trustedProxyCIDRs covering the proxy) |
Everyone shares one rate limit, and SIGNUP_BLOCKED_COUNTRIES sees the proxy’s address. |
| The identity provider reachable from the pod | Company sign-in fails at the redirect; the pod’s log says why. With signIn.email on, emailed codes still work. |
Resend reachable (when signIn.email is on) |
No sign-in codes and no notices. Company sign-in is unaffected. |
| A role that owns the database | The first start fails to build the schema (permission denied); the log says so. |
| An empty database owned by another application | The start refuses: the users table exists but is not ours. Give Simple Hack its own database. |
| The volume | Nothing starts (persistence.enabled: false is refused). |
| A second replica | Refused by design: one pod, Recreate. Uploads, imports and rate limits are in-process, and the volume is ReadWriteOnce. |
| Egress to the internet | Fine for an internal event with company sign-in, as long as the identity provider is reachable. Team sites that call outside services do that from the browser, not the pod. Site-storage webhooks (team notifications) need egress to wherever they point. |
The connector screens (MCP_APP_*) |
Off here; they are the hosted service’s. Everything else over MCP works at https://hack.example.com/mcp. |
Troubleshooting
kubectl -n simple-hack logs deploy/simple-hackhas every startup reason: a missing setting, a database it cannot reach (it waits two minutes, then exits), a schema it refuses.kubectl -n simple-hack exec deploy/simple-hack -- /usr/local/bin/simple-host migrate -statuslists which migrations the database has.- A pod that is Running but not Ready cannot ping Postgres:
/readyzreturns 503 with{"status":"unready"}. kubectl -n simple-hack port-forward deploy/simple-hack 8090:8090andcurl -H 'Host: hack.example.com' http://127.0.0.1:8090/reaches the pod without the load balancer, to tell its problems from the proxy’s.deploy/kubernetes/tests/kind-check.shruns the whole thing against a throwaway kind cluster with its own Postgres, as a release check.
Password sign-in is enabled by default in the chart (signIn.password.enabled: true, signIn.password.selfSignup: true). It renders PASSWORD_SIGNIN=on and PASSWORD_SELF_SIGNUP=on, and needs neither OIDC nor Resend. Set selfSignup: false for admin-created accounts only, or enabled: false to require another sign-in method. Admin Users creates accounts and resets temporary passwords.