Building a multi-tenant tunnel service in four days.

21tunnel started the week with a working transport and no dashboard. Four days later it had signup, email verify, hybrid approval, org-scoped RBAC, MFA, session rotation with theft detection, Stripe Checkout, and a superadmin console. This is a build-log post — what shipped each day and the three or four decisions that made the velocity possible.

What we built

The shape of the MVP, from the design doc:

• Hybrid self-serve signup → org pends → superadmin approves.
• JWT access (15 min) + opaque refresh cookie (30 day, rotating, HttpOnly, SameSite=Lax).
• Optional TOTP MFA + 10 one-shot recovery codes, hashed with argon2id.
• Owner / admin / member / viewer RBAC per org; a single is_superadmin bit that bypasses org checks.
• Stripe Checkout + Customer Portal + HMAC-verified webhook.
• Every mutating action writes an audit-log row, partitioned by month.

Week 1 — auth foundations

Day one was the bottom of the stack: argon2id password hashing, opaque-token generation, SHA-256 at-rest storage, JWT issue/verify, and repositories for sessions + refresh_tokens + email_tokens.

crates/qnt-core/src/password.rspub fn hash_password(password: &str) -> Result<String, PasswordError> {
    let salt = SaltString::generate(&mut OsRng);
    Argon2::default()
        .hash_password(password.as_bytes(), &salt)
        .map(|h| h.to_string())
        .map_err(|e| PasswordError::Argon2(e.to_string()))
}

pub fn generate_opaque_token() -> String {
    let mut bytes = [0u8; 32];   // 256 bits
    OsRng.fill_bytes(&mut bytes);
    URL_SAFE_NO_PAD.encode(bytes) // 43-char base64url
}

The JWT module is short on purpose — jsonwebtoken handles HS256 and expiry validation, we just carry the minimum needed in claims:

crates/qnt-server/src/web_auth.rspub struct AccessClaims {
    pub sub:  Uuid,     // user
    pub org:  Uuid,     // active org at login
    pub role: String,   // owner | admin | member | viewer
    pub su:   bool,     // is_superadmin
    pub jti:  Uuid,     // session_id — the middleware revokes by this
    pub exp:  u64,
    pub iat:  u64,
}

Refresh-token rotation

The refresh-token table carries a rotated_to foreign key that chains to its replacement. Presenting an already-rotated token revokes the whole session — free theft detection, no heartbeat needed:

// session.rs — rotate_refresh_token
let (old_id, session_id): (Uuid, Uuid) = sqlx::query_as(
    r"SELECT id, session_id FROM refresh_tokens
       WHERE id = $1
         AND rotated_to IS NULL
         AND revoked_at IS NULL
       FOR UPDATE",
)
.bind(old_token_id)
.fetch_optional(&mut *tx)
.await?
.ok_or_else(|| DbError::NotFound("refresh_token(rotation)".into()))?;

// insert new token, mark old one rotated_to the new id
// if this query returned zero rows, the handler calls revoke_session(...)
// — the thief and the legit user both lose access.

The cookie itself is HttpOnly, SameSite=Lax, scoped to /auth/refresh so it only flies on the one endpoint that needs it. Secure is config-gated: on by default, off only in HTTP dev.

Week 2 — RBAC + superadmin

Migration 0004 added the five core tables (sessions, refresh_tokens, email_tokens, mfa_recovery_codes, billing_events, usage_rollups) plus two columns: users.is_superadmin and organizations.approval_status. Intentionally flat — no separate superadmin table, no separate admin app. Superadmin is a boolean bit on the existing user row.

Why a bit? Two reasons:

1. Same login form. A superadmin types the same email + password + TOTP as any other user. No second authentication surface to harden separately.
2. The JWT carries it. The middleware's AuthContext has is_superadmin: bool; a single if ctx.is_superadmin { ... } branch replaces every org-membership check.

The RBAC matrix is enforced by one extractor (RequirePermission) and one ordinal helper (role_level). No per-route hand-written checks, which means the permission matrix in the design doc is also the enforcement surface. Change the matrix, update one helper.

Week 3 — the dashboard

The dashboard is a static Next.js export served from nginx. Auth state lives in a Zustand store; the access token is in-memory only (not persisted), while user + activeOrg + isSuperadmin persist to localStorage — enough to render the shell optimistically on page reload without a flash of unauthenticated UI.

The fetch wrapper does one thing well:

frontend/src/lib/auth.ts// If a request returns 401 AND we aren't already on /auth/refresh,
// try refresh once. Dedupe concurrent refreshes.
if (resp.status === 401 &&
    path !== "/auth/refresh" &&
    path !== "/auth/login") {
  const fresh = await refreshAccessToken();
  if (fresh) resp = await doFetch(fresh);
  else {
    useAuth.getState().clear();
    const returnTo = encodeURIComponent(location.pathname + location.search);
    location.href = `/login?return_to=${returnTo}`;
  }
}

Concurrent 401s share one in-flight refresh Promise (refreshInFlight) so a dashboard mount with five parallel queries doesn't rotate the token five times and race the theft-detection path. Small detail, real bug avoided.

Week 4 — dual_auth, MFA, Stripe

Two problems on day four: (a) the existing tunnel/token/event routes were still gated by a static admin bearer — nginx was no longer injecting it, so logged-in tenants got 401 on every dashboard call. (b) we wanted MFA and Stripe, but couldn't break the ops scripts that still use the static bearer.

One middleware solved both:

api/middleware_jwt.rspub async fn dual_auth(/* ... */) -> Response {
    // Fast-path: matches the static admin bearer → synthetic superadmin ctx
    if presented_matches_static_admin_bearer {
        req.extensions_mut().insert(AuthContext {
            user_id:       defaults::default_user_id(),
            active_org:    defaults::default_org_id(),
            role:          "owner".into(),
            is_superadmin: true,
            session_id:    Uuid::nil(),
        });
        return next.run(req).await;
    }
    // Otherwise fall through to JWT verification.
    require_user_auth(Extension(state), req, next).await
}

Handlers read ctx.active_org uniformly. Admin bearer → default org's view. Tenant JWT → tenant's view. One middleware, no branching in handlers.

MFA without external deps

TOTP via the totp-rs crate. Recovery codes are 10-byte random, formatted xxxxx-xxxxx-xxxxx, argon2-hashed before storage, shown once. The login flow short-circuits on mfa_enabled:

// /auth/login
if user.mfa_enabled {
    // Stateless challenge: a short JWT with role="mfa_challenge", 5-min TTL.
    return Json(json!({
        "mfa_required": true,
        "challenge_token": issue_mfa_challenge(user.id, &enc)?
    }));
}
// Otherwise: session + access_token + refresh cookie as usual.

The challenge itself is a signed JWT — no DB row for the in-flight login. /auth/login/mfa verifies it, accepts either a TOTP code or a recovery code, and issues the real session. Servers stay stateless between the two legs.

The TOTP secret is sealed at rest with an XOR-SHA256 stream keyed on the master key — not defense-grade AEAD yet, but enough that a DB leak doesn't directly hand out live TOTP seeds. AEAD upgrade is on the follow-up list.

What's next

The MVP is feature-complete. The outstanding items for a public launch:

• HTTPS + login.21tunnel.com — DNS record, Certbot, secure_cookies = true. Runbook is already written.
• Resend sending-domain verification — signup emails currently log tokens at INFO when Resend isn't configured; fine for dev, not for real users.
• Stripe test-mode → live-mode cutover — the code is done, waiting on the account.
• Nightly metered-bandwidth reporting — schema (usage_rollups) is ready; the job that reports to Stripe's metered-billing endpoint is the last missing piece.
• AEAD for the MFA secret — swap the stream cipher for chacha20poly1305.

We build in the open and write the decisions up here as we go. Every commit referenced in this post is real, tagged with its SHA — the engineering blog is where we show our work.

---