# MUST OBEY — Mobile-first build requirements
This app's PRIMARY surface is a mobile phone. Build it impeccably on mobile FIRST, then verify on tablet and desktop. Treat the rules below as non-negotiable hard constraints, not suggestions.
## Viewports to verify (every screen, every state)
- 320 px, 360 px, 375 px, 390 px, 414 px, 480 px
- 768 px, 834 px (iPad portrait / Pro 11)
- 1024 px, 1280 px, 1440 px, 1920 px, 2560 px
- Plus: 200% browser zoom, landscape orientation on every mobile width, iPhone with safe-area insets visible
## Hard layout rules
- Mobile-first CSS. Default styles target mobile; `@media (min-width: ...)` for larger viewports.
- Use `dvh` and `svh` instead of `vh` for full-height surfaces (iOS Safari URL-bar bug).
- Use `clamp()` for fluid typography across all viewports.
- Prefer container queries (`@container`) over media queries for component-level responsiveness.
- Use `min(100%, ...)` widths so content never overflows. Zero horizontal overflow at any viewport.
- Add `` to every page.
- Apply `padding: max(safe-area-inset-X, fallback)` on every edge-bleeding container so notched iPhones in landscape never clip content.
- Wide tables and code blocks scroll INSIDE their container (`overflow-x: auto`), never push the body.
- Use `background-attachment: scroll` on mobile, not `fixed` (iOS Safari repaint bug).
- Avoid `backdrop-filter` on animated elements. Use it sparingly on static surfaces only.
- **Canvas Scaling**: Canvases must dynamically scale with window resize events and properly handle high-DPI screens (`window.devicePixelRatio`). Set physical dimensions (`canvas.width`/`canvas.height`) using pixel ratio and render relative to this grid, using CSS to control responsive viewport scaling.
- **Robust Storage**: Every access to `localStorage`/`sessionStorage` (especially `JSON.parse` of loaded state or writes) MUST be wrapped in a `try-catch` block to handle disabled storage, private browsing mode, quota limits, or corrupted JSON gracefully. Fall back to a robust in-memory object store.
## Touch & accessibility
- Tap targets ≥ 44 × 44 px on touch (Apple HIG). Increase to 48 px under `@media (hover: none) and (pointer: coarse)`.
- All interactive controls reachable by keyboard with a visible focus ring; respect `:focus-visible`.
- Color contrast ≥ 4.5:1 for body text, 3:1 for UI components.
- All images have meaningful `alt`. Decorative images use `alt=""`.
- Respect `prefers-reduced-motion: reduce` — zero animation durations under that query.
- Forms validate inline; error messages are specific, not "Invalid input".
- Modals: focus trap, `Esc` closes, `role="dialog"`, `aria-modal="true"`, focus restored on close.
## Performance bar (Lighthouse mobile, throttled 3G/4G)
- LCP < 2.5 s · INP < 200 ms · CLS < 0.1
- JS bundle gzip < 200 KB mobile-first; lazy-load non-critical screens via `React.lazy` / dynamic imports.
- No render-blocking resources above the fold.
- Images: WebP/AVIF preferred, `loading="lazy"`, explicit `width`/`height` attributes (zero CLS), `srcset` for retina.
- Videos: `preload="metadata"`, low-resolution poster, max 720p mobile fallback. Never autoplay with audio.
- Fonts: `font-display: swap`; preload only the one used above the fold.
- Smooth scroll honoured via CSS `scroll-behavior: smooth` with reduced-motion fallback.
## Pre-ship mobile checklist (the deployer MUST verify before declaring done)
1. Open at 375 px in DevTools — every screen scrolls vertically only; zero horizontal scroll.
2. Browser zoom 200% — layout reflows without overlap.
3. iPhone Safari with the URL bar visible AND landscape — no content under the home indicator; no notch clipping.
4. iPad portrait (768 px) and landscape (1024 px) — no awkward gaps; tablet-specific breakpoints land cleanly.
5. Tap every interactive element with a thumb at real-device size — every target is easy to hit.
6. `prefers-reduced-motion: reduce` — every transition / animation skips cleanly, scroll-behavior becomes instant.
7. Lighthouse mobile score ≥ 90 across all 4 categories.
8. Zero `console.error` and zero CLS shift in real-device testing on a mid-tier Android (e.g. Pixel 6a) and an iPhone SE.
---
The original template starts below. All rules above apply on TOP of whatever this template specifies.
---
# Live Tutor
## 1. Project
**Live Tutor** is a calm voice that walks you through whatever you
are stuck on — homework, a piece of code that won't compile, a
recipe step you have never tried, an Excel formula returning
`#REF!`, an IKEA instruction with the diagram missing a step, the
lawnmower that won't start — by watching what your camera sees and
asking you Socratic questions until you arrive at the answer
yourself. It does not give you the answer. It asks what you have
already tried, prompts the next move, waits while you do it, and
celebrates when you get there.
Drop the phone in a stand, tap **Start a session**, point the
camera at the problem on the table — a calculus exercise on paper,
a `useEffect` infinite loop on your laptop screen, a stuck
carburettor float bowl with screws half-out, a wobbly Billy
bookcase with cam-locks loose — and a Live API session opens. The
model SEES what the camera sees and ASKS what you have already
tried. You explain in twenty seconds. It asks the next Socratic
question, anchored to what is actually on screen. You answer.
Conversation continues. At the end the session is saved as a
transcript with the photographed work, the steps you took, and the
moment the model heard you arrive.
This is the app a high-school student opens at the kitchen table
when their parent has gone to bed and the calculus problem on the
worksheet still has not given up its u-substitution. It is the app
a junior developer opens at 11pm when a React hook is firing in an
infinite loop and the Stack Overflow rabbit-hole has dead-ended. It
is the app a new-flat dweller opens with the IKEA Billy bookcase
sprawled across the living-room floor and a leftover cam-lock that
the diagram does not show. It is the app a grandparent opens when
the inherited bread recipe says "knead until the windowpane test"
and they have never heard of the windowpane test. Same Socratic
shape, very different shaped problems.
**The 30-second demo that proves the magic:** the user puts the
phone in a stand, taps **Start**, points the camera at a
handwritten calculus problem — `∫ x · sin(x²) dx`. The model's
voice (calm, unhurried, with the post-I/O 2026 acoustic nuance):
*"Walk me through what you've already tried."* The user explains
for twenty seconds: tried integration by parts, got messy, suspects
substitution. The voice: *"Good — what would you pick as `u`?"*
The user thinks aloud, writes `u = x²` on the paper, holds it back
up. The voice sees it and responds: *"That's the choice I'd make.
What's `du`?"* The user works out `du = 2x dx` on the paper. The
voice: *"You've got `2x dx` on the right and `x dx` in the
integral — what does the integral become in terms of `u` and `du`,
once you handle that factor of 2?"* The user works it out: `½ ∫ sin(u) du`.
The voice: *"That's it. Now finish."* The user writes
`-½ cos(u) + C` and then `-½ cos(x²) + C`, holds it up. The voice
sees the final answer, holds for a second, and says quietly: *"That's
right. You did the substitution; I just kept you company. Want me to
save the transcript?"*
The user walks away with the problem solved by *them*. The
transcript saves to a quiet feed — the photograph of their work,
the Socratic dialogue beside it, the time the session took, the
hint count (zero this time), and the moment the model heard them
arrive.
**Tagline:** _Point your camera at what you're stuck on. Get a
calm voice that asks the next question — never gives the answer._
## 2. Target audience
The job-to-be-done is universal: *"I am stuck on a thing and I
want someone to walk me through it without robbing me of the
satisfaction of solving it."* That job shows up everywhere.
- High-school and university students stuck on a problem set —
algebra, calculus, physics, chemistry stoichiometry, organic-
chemistry mechanisms, statistics, micro-economics graph problems
- Junior developers debugging code — a React hook in an infinite
loop, a Python list-comprehension returning the wrong shape, a
Rust borrow-checker error, a SQL join with NULLs in the wrong
places, an `await` that should have been a `Promise.all`
- Adults learning new manual skills — a sourdough recipe at the
"windowpane test" step, the gluten-free conversion of a familiar
cake, the first time deboning a chicken thigh, the first time
blanching tomatoes for sauce, knife sharpening on a whetstone
- Excel / Sheets users with a broken formula — a `#REF!`, an
`#N/A` in a VLOOKUP that should not be there, an array formula
returning a single value, a pivot table double-counting
- DIY at home — assembling IKEA furniture with a confusing diagram,
re-grouting a bathroom tile, hanging a picture without a stud-
finder, replacing a flapper valve in a leaky toilet cistern,
resetting a tripped RCD on a consumer unit
- Small-engine and appliance repair — the lawnmower that won't
start, the dishwasher with a clogged drain pump, the washing
machine with the F08 error code, the bicycle derailleur that
won't shift into top gear
- Music students working out a chord voicing, a fingering, a
bowing — guitar, piano, violin, ukulele, traditional
instruments
- Language-learners reading a real-world sign, menu, or document
they want to understand without having it translated wholesale
- Anyone facing an instruction sheet, a manual, a recipe, a tax
form, a knitting pattern, a sewing pattern, an LED-strip wiring
diagram, an electrical-panel label — they want to be *taught*
through it, not given the answer
- Parents helping kids with homework who want to model "ask
questions, don't give answers" — the app demonstrates the
posture they want to learn themselves
## 3. Core value propositions
Surface these clearly through copy, visual emphasis, and section
ordering — they are the reasons the user picks this over the
"give me the answer" alternatives.
- **The camera is the input.** You do not type a problem into a
textarea. You do not paste an error. You hold the phone over
the paper, the laptop screen, the carburettor, the bookcase, and
the model SEES what you see. Live API on Gemini 3.1 Flash Live
takes the video frames at one frame per second and the audio
stream together, and reasons across them at conversational
latency.
- **Socratic by design.** The model never solves the problem in
one shot. It asks what you have already tried. It prompts you
to the next step. It waits while you work. It celebrates when
you arrive. The system instruction enforces this; the UI
reinforces it; the hint budget visible at the top of the
session keeps it honest.
- **Real-time vision + voice.** Post-I/O 2026 the Live API's
acoustic nuance is the headline — pitch and pace that feel
human, not robotic. Combined with the video-frame channel, the
experience is "a calm tutor leaning over your shoulder", not "a
chatbot reading a transcript". This is the capability that did
not exist before May 2026.
- **One hint, then another, never the answer.** A hint budget
(default 3 per session) is visible at the top. The model will
not exceed it without you explicitly tapping *"I'm really
stuck — give me a bigger hint"*. Even then it gives you a
bigger hint, not the answer. The Socratic discipline is a
feature, not a side-effect.
- **The transcript is the artefact.** At the end of every session
you have a saved record: the photograph of your work, the
Socratic dialogue beside it, the timestamps, the hint count,
the moment you arrived. Reviewable later. Shareable with a
classmate. Searchable across all your sessions ("when did I
last work on integration by substitution?").
- **For adults learning, the rule is the same.** Whether it is a
calculus problem or a carburettor float bowl, the model never
paste-solves. For code the user types the next line themselves
— the model describes what to type, not what to copy. For a
recipe the user does the step themselves — the model describes
the texture/colour/feel to look for. The dignity of solving
it yourself is preserved across domains.
- **Honest about uncertainty.** If the model cannot see the
problem clearly through the camera (glare, focus, partial
view), it says so calmly and asks you to adjust — it does not
guess. If the problem is outside its competence (a graduate-
level proof, a specialised industrial machine), it says so
rather than fabricating a method.
- **Hard rule on professional advice.** The tutor mode is for
learning. Anything that smells like medical, legal, electrical-
safety, gas-safety, or structural-safety advice surfaces a
not-a-substitute banner. The model can ask the user what they
see and ask them what they have considered — it does not
diagnose, it does not authorise, it does not say "this is
safe". The disclaimer is not an afterthought; it is part of
the contract.
## 4. Features to build
- **Start a session** — one big record button on the home screen;
picks up the device camera and microphone, opens a Live API
websocket session with `gemini-3.1-flash-live-preview`, streams
audio in (PCM 16k) and video frames at 1 FPS, plays audio out
(PCM 24k) through the device speaker or the user's headphones.
- **Subject hint chips** at session start — *Maths*, *Code*,
*Recipe*, *Spreadsheet*, *DIY / repair*, *Music*, *Language*,
*Other*. Each chip nudges the session-system-prompt's domain
vocabulary; none of them changes the Socratic posture.
- **Hint budget visible at the top** — three filled dots by
default; one dot empties each time the model offers a hint;
when zero are left, the model defaults to encouragement rather
than additional hints, and the user can tap *"I'm really
stuck"* to request a bigger (but still not full-answer) hint.
- **Live transcript pane** below the camera preview — caption
streams the model's spoken words in real time; the user's
spoken words appear in a different style (less prominent, the
user does not need to read their own words). The transcript is
scrollable mid-session if the user wants to reread.
- **"Show me what you see"** toggle — when on, the model
occasionally pauses to describe what it sees in the camera ("I
see a handwritten integral, `∫ x · sin(x²) dx`, and you've
written `u = x²` in the margin"). Useful for blind / low-vision
users; also useful when the user wants the model to confirm
framing before they start working.
- **"I'm stuck" button** — taps add one hint to the budget for
this session only (max +2 over the session). The model
acknowledges the extra latitude calmly and gives a slightly
bigger hint, never the answer.
- **"I think I've got it" button** — the user taps when they
believe they have the answer; the model asks them to read it
out loud or hold up their working, and confirms or asks a
follow-up question. This is the "moment of arrival" the
transcript marks.
- **Pause / Resume** — clean pauses on the Live API session
(closes the audio channel but preserves session state for up
to two minutes). Useful when the user needs to write quietly
without the model listening.
- **End session + save transcript** — closes the Live API
session; the server writes the transcript to Firestore, the
final photograph (last frame the model commented on) to
Firebase Storage, the hint count, the duration, the moment-of-
arrival timestamp.
- **Session library** — a chronological feed of past sessions,
each one a card with the subject, the photograph of the work,
the first line of the transcript, the moment-of-arrival, and
the hint count. Searchable.
- **Session detail view** — full Socratic transcript beside the
photograph(s) of the user's working. Each model line
timestamped. Each user line timestamped. The hint-budget
history is shown as a small timeline at the bottom ("3 hints
remaining → 2 hints remaining at 1:42 → 2 hints remaining,
moment of arrival at 4:18").
- **Resume an unfinished session** — if the Live API session
hit the 2-minute audio+video cap before the user arrived, the
session is saved as "in progress" and can be resumed with a
new Live API connection plus a short text-only summary of
where the previous segment left off (passed as system
context).
- **Subject-specific affordances:**
- *Code mode*: the camera frames a code editor; the model
reads what is on screen and asks the user to type the next
line themselves — never pastes a solution. The transcript
captures the code snippets verbatim from the OCR.
- *Recipe mode*: the model describes textures and colours
rather than minutes ("the dough should pass the windowpane
test — stretch a small piece thin between your fingers; if
you can see light through it without tearing, you're
there").
- *DIY mode*: the model surfaces a "not-a-substitute" banner
whenever the user shows electrical work above 12V, gas
components, load-bearing structural work, or anything with
safety implications. The model asks; it does not authorise.
- *Maths mode*: the model resists arithmetic temptations —
it never computes; it asks the user what they get.
- **Reduced-motion + calm-acoustic mode** — toggle in settings;
reduces the model's pace by ~15% and removes intonation
flourishes, for users who find expressive TTS distracting.
- **Family-mode safety wrapper** — when enabled, the session
refuses to engage with any subject outside *Maths, Code,
Recipe, Spreadsheet, DIY, Music, Language* and refuses any
off-topic conversation, so a kid using the parent's device
stays in the homework lane.
- **Practice session library** — a small built-in library of
starter problems per subject (one algebra problem, one
React-hook bug, one sourdough step, one Excel formula error,
one IKEA bookcase, one bicycle derailleur). Useful for the
first-visit demo, useful for parents wanting to see the
posture before handing the phone to a kid.
- **Export transcript** — Markdown export for code sessions, PDF
export for maths sessions (with the photograph of the working
inline), email-to-self export for any session.
- **Calendar reminders** — local reminders only, never push from
the server. "Revisit the calculus session from Tuesday before
the Friday quiz" is the right shape.
## 4b. Required Gemini capabilities + backend services
**This template's intelligence comes from the Gemini capabilities
below. Wire them up explicitly — don't substitute generic LLM
calls.**
### Gemini capabilities (the load-bearing intelligence)
- **Live API session** (`gemini-3.1-flash-live-preview`) — the
hero capability. Streams audio in (PCM 16k mono) and video
frames at ≤1 FPS; streams audio out (PCM 24k mono); takes
text turns mixed into the same channel for things like the
session preamble. Acoustic nuance (pitch, pace) was the
I/O 2026 lift — sentences sound considered, not robotic. **Hard
limits**: 15 min for audio-only sessions, **2 min for audio +
video sessions**. Function calling is **synchronous only** —
the model pauses speaking while a function returns. The
session can be resumed with ephemeral session tokens.
- **Live API function calling** — three synchronous functions
are wired:
- `markHint()` — the model declares it just offered a hint;
the client decrements the hint budget and re-renders the
dots. The model must call this any time it gives more than
a Socratic question, so the user sees the budget shrink.
- `markArrival(answerSummary, confidence)` — the model
declares that the user has arrived at the answer; the
client logs the moment-of-arrival timestamp, prompts the
user to hold up their work, and offers the *Save
transcript* action.
- `requestStillFrame()` — the model asks the client to grab
a high-resolution still (1080p) of the current camera view
because the 1-FPS stream is too compressed to read fine
detail (handwriting, code, recipe text, error codes). The
client uploads the still to the Gemini Developer API Files
API and the model references the resulting `files/*`
resource name (e.g. `files/abc123xyz`) via `fileData.fileUri`
in the next turn.
- **Pre-session subject vocabulary primer** (Gemini 3.5 Flash,
thinkingLevel `low`) — when the user picks a subject chip, a
short text-only call assembles a `SessionPreamble` JSON that
the client prepends as the first system message into the Live
API session. The preamble names the subject's vocabulary, the
domain-specific Socratic patterns, and any safety flags
(e.g. "if user shows mains electrical, surface disclaimer").
- **Transcript post-processor** (Gemini 3.5 Flash, thinkingLevel
`low`) — after a session ends, takes the raw turn-by-turn
transcript + the captured still frames and emits a clean
`SessionTranscript` JSON: the Socratic dialogue, the hint
events, the moment-of-arrival, the topic summary, the next-
session-suggestion (one short follow-up problem the user
could try). Structured output via `responseSchema`.
- **Subject classifier** (Gemini 3.5 Flash, thinkingLevel
`low`) — if the user does not pick a subject chip and just
starts a session, the first ~10 seconds of video frames go to
a classifier call that returns one of `maths`, `code`,
`recipe`, `spreadsheet`, `diy`, `music`, `language`, `other`
plus a confidence score. The result feeds the preamble. If
confidence < 0.6 the client shows the chips to ask explicitly.
- **Image still understanding for the transcript** (Gemini 3.5
Flash, multimodal image) — the Live API stills captured via
`requestStillFrame()` are also fed back into the post-
processor so the transcript can include readable OCR of code,
legible maths working, and recipe text. Used post-session,
not in the live loop (the live loop has Live API's own
vision).
- **Subject-mode safety classifier** (Gemini 3.5 Flash,
thinkingLevel `low`) — runs on every still frame the model
captures via `requestStillFrame()`; returns a small JSON
flagging whether the still shows mains-electrical, gas,
load-bearing structural, medical/wound, firearm, or other
professional-territory content. If flagged, the client
surfaces the not-a-substitute banner and the next Live API
turn injects a system message reminding the model not to
authorise the work.
- **Hero / empty-state illustration** (`gemini-3.1-flash-image`,
Nano Banana 2) — generates the welcome and empty-state
imagery: a phone in a stand on a kitchen table at golden hour
with a notebook open showing a calculus problem, soft warm
light, no people.
- **Demo voice samples for the audio-preview** — recorded once
via `gemini-3.1-flash-tts-preview` and cached as static MP3
in `/public/audio-samples/`; the landing page plays them so a
visitor can hear the tutor voice before starting a real Live
API session. **Do not use 2.5-flash-preview-tts** — that is
the pre-I/O preview and has been superseded.
- **Thinking levels** — `low` for every text-only call in this
template (subject classifier, preamble, transcript post-
processor, safety classifier). The Live API model
(`gemini-3.1-flash-live-preview`) does not take a
`thinkingConfig` field — omit it entirely. The TTS and image-
generation calls also do not take `thinkingConfig`.
### Backend services
- **Auth — Required.** Firebase Auth with Google sign-in (auto-
provisioned by AI Studio Build). Apple sign-in is optional and
user-configured (requires Apple Developer account, Service ID,
Key ID, private key in the Firebase Auth console). Magic-link
email (used to email a saved transcript to the user) requires
the sender domain to be authorised in Firebase Auth.
- **Database — Required.** Firestore for `users`, `sessions`,
`transcripts`, `still_frames_metadata`, `practice_problems`,
`family_mode_settings`. Sessions are private to the owner by
default.
- **File storage — Required.** Firebase Storage for the captured
still frames (the `requestStillFrame()` images) and for any
recorded audio the user chooses to save. **Storage is NOT
auto-provisioned by AI Studio Build today** — enable it in the
Firebase console and wire the bucket name into the AIS Build
project before the first session runs. Pre-signed URLs only.
- **Live API gateway — Required.** A Cloud Run server function
(auto-provisioned) opens the Live API websocket connection
using the project's `GEMINI_API_KEY` from Secrets Manager and
proxies frames + audio between the client and Google. **Do not
expose `GEMINI_API_KEY` to the browser.** Use ephemeral session
tokens issued by the gateway so the client only ever holds a
short-lived token, not the API key itself. The Cloud Run free
2-app deploy (post-I/O) covers this app's gateway out of the
box.
- **Email — Required (transactional).** Magic-link share email
for transcripts the user chooses to email to themselves or to
a classmate. Sender domain authorised in Firebase Auth.
- **Payments — Not needed for v1.** Free for personal use.
Future "Family Pro" tier (multi-child sub-accounts, parent
digest emails, longer hint budgets per subject) might charge
via Stripe; not built in v1.
- **Workspace integration — Optional.** Post-I/O 2026 Workspace
integration without OAuth handshake — for users who want
transcripts auto-saved to Google Docs / Drive, this is a one-
toggle setting and no longer requires a per-user OAuth
consent flow. Verify availability in your AIS Build
configuration before promising it.
- **External APIs:** Gemini API for all intelligence (Live API,
Flash text calls, image generation, TTS for cached samples).
No required external API beyond Gemini.
**Environment variables:** every secret (Gemini API key, Firebase
service-account JSON, Stripe key if pro tier added) lives in
environment variables — never in client bundle. Include a
`.env.example`.
**Auth + data privacy reminders:** never log secrets · never store
passwords in plain text · use HTTPS everywhere · honour 'delete my
account' inside the UI · explicit opt-in for any analytics · the
user's session audio, video frames, and transcripts are never sent
to Gemini for model training (use the Gemini API on the paid tier,
where Google does not use your content for model training, per the
Gemini API Additional Terms) · family-mode sessions are scoped to
the family unit and never indexed.
**Read this first — prompt-craft rules that apply to every call in
this template:**
1. **Name the model variant explicitly** in every Gemini API call.
Do not let the agent pick the model. See the per-call matrix
below.
2. **Pin `thinkingLevel` explicitly** per call. See the matrix.
Models that don't take a thinkingConfig (the Live API model,
the image-generation model, the TTS model) — omit the field
entirely.
3. **Seed the JSON Schema as a fenced TypeScript / Zod block** in
the system instruction or `responseSchema` field. The literal
schemas are below. **Convert the Zod schema to Gemini's
`Schema` type via the SDK helper** before passing to
`responseSchema` — do NOT pass raw Zod. **Numeric `min`/`max`
constraints are documentation only inside `responseSchema`;
clamp on the server after the response arrives.**
4. **Pin the system instruction separately** from user input. Use
the `systemInstruction` field for persona + behavioural rules;
use `contents` for user input. Never concatenate.
5. **Pre-declare tools as an enable/disable list** per call. The
matrix below names which tools are enabled per call. Tools
NOT listed for a call should be disabled.
6. **State negative constraints explicitly** — they are listed
below. They are NOT "be careful" suggestions; they are hard
rules the model must follow.
7. **Live API function calling is synchronous only.** The model
pauses speaking while a function call returns. Keep function
handlers fast (<200ms) — long ops should be deferred to a
post-turn worker, not blocked on inside `markHint` or
`markArrival`.
8. **Live API video+audio sessions cap at 2 minutes.** Build
resumption explicitly: at 1:45 the client warns the user, at
1:55 the client gracefully closes and queues a resume token.
9. **Files API uses `files/*` resource names, not `gs://` URIs.**
The AI Studio Build runtime uses the Gemini Developer API
(`@google/genai` SDK). Files API `upload` returns a resource
name of the form `files/abc123xyz`, passed via `fileData:
{ fileUri, mimeType }`. `gs://` URIs belong to Vertex AI /
Cloud Storage — a different surface, not accepted here.
10. **Grounded responses can wrap JSON in ```json fences or add
prose preamble** (the next-session-suggestion call uses
`google_search`). Server-side, strip fences and brace-extract:
```typescript
function safeExtractJSON(raw: string): T {
const clean = raw.replace(/```json\s*|```/gi, '').trim();
const s = clean.indexOf('{'); const e = clean.lastIndexOf('}');
if (s === -1 || e === -1) throw new Error('No JSON boundaries in grounded response');
return JSON.parse(clean.slice(s, e + 1)) as T;
}
```
11. **Strip unsupported Zod modifiers before passing to
`responseSchema`** — Gemini's OpenAPI subset rejects `.regex()`
/ `pattern`, fixed-length `z.tuple()`, and other custom
validators. Use a sanitizer that flattens tuples to arrays and
removes regex patterns before serializing. Validate those
constraints in middleware AFTER parsing.
### Per-call model + tools matrix
| Call | Model | thinkingLevel | Tools enabled |
|------|-------|---------------|---------------|
| Live session (audio in + video frames in + audio out) | `gemini-3.1-flash-live-preview` | n/a | function calling: `markHint`, `markArrival`, `requestStillFrame` |
| Subject classifier (first ~10s of frames) | `gemini-3.5-flash` | low | (none) — pure multimodal image input |
| Session preamble assembly (subject → SessionPreamble JSON) | `gemini-3.5-flash` | low | (none) |
| Still-frame safety classifier (per captured still) | `gemini-3.5-flash` | low | (none) |
| Transcript post-processor (raw turns + stills → SessionTranscript) | `gemini-3.5-flash` | low | (none) |
| Next-session suggestion (one follow-up problem) | `gemini-3.5-flash` | low | `google_search` grounding (no `responseSchema` on this call — see note) |
| Hero / empty-state illustration | `gemini-3.1-flash-image` (Nano Banana 2) | n/a | n/a |
| Cached voice samples for the landing page | `gemini-3.1-flash-tts-preview` | n/a | n/a |
*Note for builders:* the Live API call does NOT take a
`thinkingConfig`; omit it. The image-gen and TTS calls also do
NOT take a `thinkingConfig`. The grounded next-session-
suggestion call cannot combine `responseSchema` with
`google_search` in the same Gemini call today — instruct the
model to emit JSON in the text body and parse it server-side;
read citation URLs from
`response.groundingMetadata.groundingChunks[].web.uri`. **Do not
use `gemini-3.5-flash`, `gemini-3.5-flash`, or `gemini-2.5-flash-
preview-tts`** — those are pre-I/O 2026 strings and now resolve
to deprecated/missing endpoints.
### Primary structured-output schemas (seed verbatim in the prompt)
```typescript
import { z } from "zod";
const SubjectMode = z.enum([
"maths", "code", "recipe", "spreadsheet",
"diy", "music", "language", "other",
]);
const HintEvent = z.object({
hint_index: z.number().min(1).max(5), // 1..3 base + up to 2 extra
given_at_iso: z.string(),
hint_size: z.enum(["small", "medium", "bigger"]),
hint_text_summary: z.string(), // one sentence, not the verbatim hint
user_was_stuck_for_seconds: z.number().nullable(),
});
const SafetyFlag = z.object({
category: z.enum([
"mains_electrical", "gas", "load_bearing",
"medical_wound", "firearm", "chemical_handling",
"child_safety", "other_professional",
]),
detected_in_still_uri: z.string(),
surfaced_disclaimer: z.boolean(),
});
const SessionPreamble = z.object({
subject_mode: SubjectMode,
vocabulary_anchors: z.array(z.string()), // domain words the model will use
socratic_patterns: z.array(z.string()), // questions to ask, never to answer
safety_watchlist: z.array(z.string()), // categories to flag if seen
initial_greeting_one_line: z.string(), // what the model opens with
});
const TranscriptTurn = z.object({
speaker: z.enum(["model", "user"]),
spoken_at_iso: z.string(),
text_verbatim: z.string(),
was_hint: z.boolean(), // true if this model turn was a hint
references_still_uri: z.string().nullable(),// if the model referenced a captured still
});
const SessionTranscript = z.object({
session_id: z.string(),
subject_mode: SubjectMode,
started_at_iso: z.string(),
ended_at_iso: z.string(),
duration_seconds: z.number(),
hint_budget_at_start: z.number().min(1).max(5),
hints_used: z.array(HintEvent),
hint_budget_remaining: z.number().min(0).max(5),
arrival_event: z.object({
arrived_at_iso: z.string().nullable(), // null if user did not arrive in this session
arrival_summary: z.string(), // one sentence describing the answer
arrival_confidence: z.number().min(0).max(1),
}),
safety_flags: z.array(SafetyFlag),
still_uris: z.array(z.string()), // Files API `files/*` resource names of stills captured
turns: z.array(TranscriptTurn),
topic_summary_one_line: z.string(), // for the session library card
next_session_suggestion: z.object({
suggestion_text: z.string(), // a follow-up problem the user could try
suggestion_source_authority: z.string().nullable(), // if grounded, the source
}).nullable(),
});
const SubjectClassification = z.object({
inferred_subject_mode: SubjectMode,
confidence: z.number().min(0).max(1),
visible_artefacts: z.array(z.string()), // what the classifier saw
needs_explicit_chip: z.boolean(), // true if confidence < 0.6
});
type SessionPreamble = z.infer;
type SessionTranscript = z.infer;
type SubjectClassification = z.infer;
```
### Common failure modes (and how to avoid them)
- **The model gives the answer.** The single most common
failure. The Live API system instruction MUST forbid giving
the answer outright; the `markHint()` function call records
every hint so the client can enforce the budget; the post-
session transcript post-processor counts hint events and flags
any session where the model crossed the budget. If the budget
was crossed, log to telemetry so the prompt can be tightened.
- **The model goes longer than 2 minutes.** Audio+video sessions
cap at 2 min on Live API. The client must warn at 1:45, save
state at 1:55, and offer **Resume** which opens a new Live API
connection with the previous session's transcript summary as
a fresh system message. Do NOT promise "continuous" sessions.
- **The Live API model picks the wrong subject vocabulary.**
Without the `SessionPreamble`, the model defaults to general-
purpose tutoring and uses awkward phrasing for code or
recipes. Always run the subject classifier (or use the user's
chip pick) and inject the preamble as the first system
message into the Live API session.
- **A safety-flagged frame slips through silently.** The
still-frame safety classifier runs on every captured still
and the client must surface the not-a-substitute banner
before the next Live API turn. The next turn's system message
must include "user is showing mains-electrical work; do not
authorise; ask what they see and ask what a qualified
electrician would check".
- **The model recomputes maths/arithmetic.** The Live API model
is a fast multimodal language model, not a calculator. It
should never compute; it should always ask the user what
*they* get. Negative constraint enforced in the system
instruction.
- **The model paste-solves code.** Forbidden. In code mode the
model asks the user to type the next line; if the user asks
"what's the line?" the model describes what the line does
("a `useEffect` that depends on the value you want to read
but not write — what would you put in the dependency
array?"), not the literal characters.
- **The transcript post-processor invents turns that did not
happen.** Hard rule: the post-processor receives the raw
turn-by-turn audio transcript from the Live API and must
only re-format / clean / summarise — never fabricate. If a
turn is unclear in the audio, mark `text_verbatim` with
`[unclear]` rather than guessing.
- **Subject classifier mislabels a recipe as DIY.** A baking
recipe shown on a tablet next to a kitchen counter can read
like DIY at low frame rates. If classifier confidence is
below 0.6 the client MUST show the chips for the user to
pick explicitly, never silently default to the wrong mode.
- **Live API ephemeral token leaks via the browser console.**
Use the Cloud Run gateway to mint short-lived tokens (e.g.
10-min TTL), bind them to the user's Firebase Auth UID, and
rotate per session. Never log them to the browser console;
never store them in `localStorage`.
- **Family-mode child wanders off-topic.** When family-mode is
on, the Live API system instruction enforces a strict
subject scope. The model must politely redirect any off-
topic input ("let's get back to the maths problem on your
paper"). If the off-topic persists beyond two redirects, the
session ends with a calm note and the budget is preserved.
- **The user is blind / low-vision and the "show me what you
see" affordance is off.** Default behaviour for accessibility
users (detected via OS-level prefers-reduced-motion + screen-
reader heuristics) is to enable "show me what you see" with
a 30-second cadence so the model speaks visual descriptions
proactively, not just on request.
### Negative constraints (hard rules)
- Do NOT give the answer outright. Never. Not even when asked
directly. The model's response to "just tell me the answer"
is to offer one more hint, then ask the user to take the
next step. If the user is genuinely stuck and the hint budget
is exhausted, the model gracefully ends the session with a
warm note rather than capitulating.
- Do NOT compute arithmetic. The model asks the user what they
get. Even simple arithmetic ("what's 2 × 3?") is the user's
to do.
- Do NOT paste solutions in code mode. The user types every
line themselves. The model describes the next line in
natural language ("the dependency array should be empty
because we only want this to run once on mount") and waits.
- Do NOT translate problems wholesale in language-learning
mode. The model asks the user what they think a word means,
prompts with a related word they might know, and confirms
when they arrive — it does not just translate.
- Do NOT authorise safety-flagged work. For mains-electrical,
gas, load-bearing structural, medical wound, firearm, or
chemical-handling work, the model asks the user what they see
and what a qualified professional would check. The not-a-
substitute banner is non-dismissable in the UI for the
remainder of the session once a safety flag fires.
- Do NOT moralise about the user's choice of work. If a user
chooses to fix their own lawnmower, the app helps them
Socratically; it does not lecture them on calling a
professional.
- Do NOT gamify learning. No streaks. No badges. No "you've
earned 500 XP". The arrival itself is the reward; the
transcript is the artefact.
- Do NOT log the user's full audio or video frames beyond the
Live API session window. The raw video frames are streamed
through the Cloud Run gateway and not persisted unless the
user explicitly chose to save audio. Still frames captured
via `requestStillFrame()` ARE persisted (they are part of the
transcript artefact).
- Do NOT use the user's session audio, video, or transcripts to
train or fine-tune any model. Use the Gemini API on the paid
tier, where Google does not use your content for model
training, per the Gemini API Additional Terms. The
capabilities-info panel says this in plain English.
- Do NOT use a robotic monotone TTS. The acoustic nuance of
Live API on 3.1 Flash Live is the headline experience —
pace and pitch should match the moment (slower and softer
when the user is stuck; brighter and faster on arrival).
- Do NOT exceed the hint budget without the explicit "I'm
really stuck" tap. The function call discipline enforces
this — every hint emits `markHint()` and decrements the
budget client-side.
- Do NOT ship without the food-safety / professional-territory
not-a-substitute banner wired up. The safety classifier runs
on every captured still; the disclaimer is required, not
optional.
### Per-call `systemInstruction` strings
Use these as the literal `systemInstruction` field for each
Gemini API call the built app makes.
### Call: Live API session (the core tutor loop)
Model: `gemini-3.1-flash-live-preview` · thinkingLevel: n/a · Tools: function calling (`markHint`, `markArrival`, `requestStillFrame`)
```
You are a calm, patient tutor. You sound like a kind older
sibling or a thoughtful coach. You are NEVER a chatbot, never a
search engine, never a stack-overflow paste. You are a voice
that asks the next question.
You can SEE what the user's camera sees (video frames at ~1
frame per second) and you can HEAR what the user says
(streaming audio). You speak back to the user in audio.
YOUR JOB:
1. When the session starts, greet the user briefly (one short
sentence) and ask what they're working on. If you can already
see what's on the camera (a maths problem on paper, a code
editor with an error, a recipe step, a piece of furniture
being assembled, a small engine with the housing open, a
musical instrument with the user's hands on it, a sign in a
foreign language), name it specifically in your greeting
("I can see the integral ∫ x · sin(x²) dx — walk me through
what you've already tried").
2. ASK what the user has already tried before suggesting
anything. Do not jump to a hint until you've heard their
approach. If the user says "nothing yet", that's fine —
suggest they start by looking at the problem and naming
what kind of thing it is (a substitution problem, a syntax
error, a missing ingredient step, a loose cam-lock).
3. Move the user FORWARD by one step at a time. Each of your
turns is a question or a small Socratic prompt, never a
solution. "What would you pick as u?" is good. "Pick u = x²"
is too much. "The substitution is u = x²; then du = 2x dx;
then the integral becomes ½∫sin(u)du" is forbidden.
4. Use the user's own vocabulary. If they say "the squiggly
thing" point to it in the camera and name it ("the integral
sign"); if they say "the parens", say "the parens"; if
they say "that bit", ask "the part where you wrote the
number?" — match their language.
5. CELEBRATE quietly when the user arrives. A single calm line
("That's it. You did the substitution; I just kept you
company"). No fireworks, no exclamation salad.
WHEN YOU GIVE A HINT (and you should give few):
- Call the `markHint()` function so the client can decrement
the visible hint budget. Pass the hint size: "small" (a
pointing-finger question), "medium" (naming the technique
but not the steps), "bigger" (walking through the first
step's setup without giving the result).
- A "small" hint is the default. Move from small → medium →
bigger only across turns, never within one turn.
- After three hints (the default budget), default to
encouragement and reflection ("you've gotten this far —
what's left to check?"). Do not offer a fourth hint
unsolicited. The client will tell you if the user has
tapped "I'm really stuck" — then you can offer one
bigger hint, but never the answer.
WHEN YOU CAN'T READ THE CAMERA CLEARLY:
- Call `requestStillFrame()` to grab a 1080p still you can
read. Wait for the client to confirm; then in your next turn
reference what you can now see.
- If the still is still unclear, say so calmly and ask the
user to reposition, improve lighting, or move closer. Do
NOT guess.
WHEN THE USER ARRIVES AT THE ANSWER:
- Confirm their answer with one Socratic check: "read it
back to me" or "hold up the page so I can see". Do not
declare arrival from a glance alone.
- When confirmed, call `markArrival(answerSummary,
confidence)` so the client can save the moment-of-arrival
timestamp. `answerSummary` is one sentence in the user's
own framing; `confidence` is your honest read of whether
they really got there.
WHEN THE WORK SHOWS SAFETY-PROFESSIONAL TERRITORY
(mains electrical, gas, load-bearing structural, medical
wound, firearm, chemical handling):
- The client will inject a system note when a safety flag
fires. From that point in the session you do NOT authorise
the work. You can still help the user think Socratically
about what they SEE and what a qualified professional
would check. Do not say "this is safe" or "this is fine
to do yourself". Do say "what would an electrician /
plumber / structural engineer look for here?".
- If the user asks you directly "is this safe", your answer
is "I can help you think through what to check, but I'm
not qualified to say it's safe — that's what an
electrician / plumber / clinician / structural engineer
is for".
HARD RULES:
- Never give the answer. Never. If asked directly, offer one
more hint and ask for the next step.
- Never compute arithmetic. Ask the user what they get.
- Never paste code. In code mode, describe what the next
line does; the user types it.
- Never translate wholesale in language mode. Prompt with a
related word the user might know.
- Never moralise. Never lecture. Never gamify.
- Never use exclamation salad ("Great job!! Awesome!! You
did it!!"). One quiet, sincere line at arrival.
- Never go off-topic in family-mode (the client will tell
you when family-mode is on). If the user wanders, redirect
calmly once. If they wander again, redirect again. If
they wander a third time, end the session warmly.
VOICE:
- Pace: unhurried; pauses where a thoughtful person would
pause; slower and softer when the user is visibly stuck;
brighter and a touch faster on arrival.
- Acoustic nuance: this is the post-I/O 2026 Live API
upgrade. Use it. Sound considered, not robotic.
- Length: most turns are one sentence. Two when a Socratic
prompt needs framing. Three only when celebrating
arrival.
If at any point you do not know how to help (a graduate-
level proof outside your competence, a specialised
industrial machine, a language you cannot parse), say so
honestly: "this is outside what I can help you with — let
me suggest where you might go next". Do not fabricate.
```
---
### Call: Subject classifier (first ~10 seconds of video frames)
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none)
```
You receive 5-10 video frames captured at 1 FPS from the
user's camera at the start of a Live Tutor session. The user
has not yet picked a subject chip. Your job is to classify
which subject mode best fits what is visible.
Subject modes:
- maths — handwritten or printed mathematics problems on
paper or a tablet (algebra, calculus, statistics, etc.)
- code — a code editor, terminal, or IDE visible on a laptop
or monitor screen
- recipe — a kitchen or cookbook scene; ingredients, mixing
bowls, dough, a printed or screen-displayed recipe
- spreadsheet — a spreadsheet visible on a laptop or monitor
(Excel, Sheets, Numbers); formulas, error codes (#REF!,
#N/A, #DIV/0!) visible
- diy — household repair or assembly; furniture being
assembled, tile work, plumbing, electrical, small-engine
parts, bicycle components
- music — a musical instrument with the user's hands on it
(guitar, piano, violin, etc.); sheet music; tabs on a
screen
- language — a foreign-language sign, menu, document, or
textbook
- other — none of the above; or genuinely ambiguous
Output the SubjectClassification JSON exactly. Set
needs_explicit_chip: true whenever confidence < 0.6.
Hard rules:
- Do NOT classify based on the user's clothing, room style,
or other irrelevant cues. Classify ONLY based on the work
artefact visible.
- visible_artefacts is a short list of concrete things you
saw ("handwritten integral with x squared", "VS Code with
a useEffect hook", "stainless mixing bowl with dough").
- If multiple subjects are present (e.g. a laptop with code
next to a maths textbook), pick the one most likely to be
the active subject and lower confidence accordingly.
- Confidence of 0.9+ requires unambiguous evidence in at
least 3 of the frames.
Output: SubjectClassification JSON. No commentary.
```
---
### Call: Session preamble assembly
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none)
```
You receive a subject mode (one of: maths, code, recipe,
spreadsheet, diy, music, language, other) and optionally a
short user-supplied note (e.g. "I'm a Year 12 student
working on differentiation"). Output a SessionPreamble JSON
that will be injected as the first system message into the
Live API session.
Fields:
- subject_mode: copy the input.
- vocabulary_anchors: 5-12 domain-specific words / phrases
the model should use in this session ("integral",
"substitution", "u-substitution", "antiderivative" for
maths; "useEffect", "dependency array", "stale closure",
"memoization" for code; "windowpane test", "autolyse",
"bulk ferment", "shaping", "proofing" for recipe / baking).
- socratic_patterns: 4-6 Socratic question patterns suited
to this domain ("What would you pick as your substitution
variable?", "What does the dependency array control in a
useEffect?", "What does the dough feel like right now?").
Patterns are templates, not full questions.
- safety_watchlist: any safety categories worth flagging if
seen in subsequent frames (e.g. for DIY mode: mains
electrical, gas, structural; for recipe mode: raw meat
handling, hot oil, knife use; for code mode: none unless
the code shows credentials).
- initial_greeting_one_line: a single short greeting the
Live API model can use to open the session (e.g. "I can
see your integral — walk me through what you've already
tried"). Keep it specific to the subject and warm.
Hard rules:
- Vocabulary anchors must be REAL terms in the domain, not
invented. If you don't know the term, omit it.
- The greeting must be specific to the subject and must NOT
give any hint about the problem itself. "I can see your
integral" is fine; "I see u-substitution will work here"
is forbidden.
- Do NOT translate vocabulary anchors. "u-substitution"
stays in English even if the user's interface is in
French.
Output: SessionPreamble JSON. No commentary.
```
---
### Call: Still-frame safety classifier
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none)
```
You receive a single still frame captured during a Live
Tutor session via the model's requestStillFrame() function
call. Your job is to flag whether the frame shows any
professional-territory subject matter where the app must
surface a not-a-substitute disclaimer and where the Live
API model must not authorise the work.
Categories to flag:
- mains_electrical — anything 50V or above, consumer-unit
panels, RCDs, wiring inside a wall plate, ring-main
components, electric showers, EV chargers
- gas — gas hobs / boilers / cylinders / regulators / flues
- load_bearing — structural walls being modified, lintels,
beams, joists being cut
- medical_wound — visible open wound, suture, blood, a
body part the user appears to be treating
- firearm — any firearm or major firearm component
- chemical_handling — caustic / acidic / hazardous chemical
containers (drain cleaner, pool acid, paint stripper)
- child_safety — a young child appears to be operating the
tool or handling the dangerous object themselves
- other_professional — anything that looks like
professional-only territory (industrial machinery,
scientific apparatus, certified equipment)
Output JSON:
{
"flags": [SafetyFlag, ...], // per the schema; can be []
"neutral_description_one_line": ""
}
Hard rules:
- Do NOT diagnose. "There is a wound visible on the user's
hand" is a description; "the user has a laceration that
needs sutures" is a diagnosis. Never diagnose.
- Do NOT guess. If you cannot tell whether the wire in
frame is mains or low-voltage, do not flag — but do
describe what you see neutrally.
- Multiple flags per frame are fine.
- surfaced_disclaimer: true means the client should show
the banner. Default true for any flag.
Output: JSON. No commentary.
```
---
### Call: Transcript post-processor
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none)
```
You receive the raw turn-by-turn transcript from a Live
API session (audio transcribed by Gemini live, with
speaker labels "model" and "user"), plus the list of
Files API `files/*` resource names of stills captured
during the session, plus the hint events recorded by the
markHint() function calls, plus the markArrival() event
if present.
⚠️ Strict Token Preservation: when transcribing or
re-rendering handwritten mathematics from a captured
still, map directly to canonical LaTeX (e.g.
`\\int x \\cdot \\sin(x^2) \\, dx`). Do NOT prose-describe
the expression and do NOT silently normalise notation. If
any expression is ambiguous, set `confidence: 'low'` +
`flagged_for_user_review: true` for that turn instead of
guessing.
Your job is to emit a SessionTranscript JSON that is the
saved artefact of the session.
What you do:
- Clean each turn's text_verbatim: remove "um" / "uh" /
audible breath markers, but PRESERVE the user's actual
words and the model's actual phrasing. Do NOT paraphrase.
- For each model turn, set was_hint: true if a markHint()
event was emitted during that turn.
- For each model turn that referenced a still, set
references_still_uri to the relevant Files API
`files/*` resource name.
- Compose topic_summary_one_line: one sentence the user
can read in the session library card ("Worked through
integration by substitution on ∫ x · sin(x²) dx — 1
hint used, arrived in 4:18").
- Set hints_used to the array of HintEvents (with
hint_index, given_at_iso, hint_size,
hint_text_summary, user_was_stuck_for_seconds).
- Set arrival_event from the markArrival() data if
present; otherwise set arrived_at_iso to null and
arrival_summary to "user did not arrive in this
session".
Hard rules:
- Do NOT fabricate turns. The transcript reflects only
what was actually said. If a turn was unclear in the
audio, mark text_verbatim with "[unclear]" and an
approximation in parens.
- Do NOT score the user. The transcript is a record, not
a report card.
- Do NOT auto-add hints that were not flagged via
markHint(). The model's discipline is the source of
truth.
- next_session_suggestion is set ONLY when arrival
happened AND the user opted in (the client will tell
you). Otherwise null.
Output: SessionTranscript JSON. No commentary.
```
---
### Call: Next-session suggestion
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: `google_search` grounding
```
You receive a SessionTranscript object for a session
that ended with arrival. Your job is to suggest ONE
follow-up problem the user could try in their next
session, of slightly higher difficulty (or covering a
nearby concept). Use google_search to ground the
suggestion in a real textbook problem, a real well-
known coding challenge, a real recipe variation, a
real DIY follow-up — never invent a citation.
Use `google_search` grounding for every suggestion.
Output the suggestion as JSON in the text body (NOT
via `responseSchema` — responseSchema and google_search
cannot be combined in the same Gemini call today). The
server will parse the JSON and read citation URLs from
response.groundingMetadata.groundingChunks[].web.uri.
JSON shape:
{
"suggestion_text": "",
"suggestion_source_authority": "",
"difficulty_note": ""
}
Hard rules:
- The suggestion must be at most one small step up in
difficulty. Do not pick a wildly harder problem just
because the user arrived.
- The suggestion must be specific. "Try a harder
integral" is not specific; "Try ∫ x² · cos(x³) dx
— same pattern, different exponent" is specific.
- Do NOT include URLs in the JSON body. Citations
come from groundingMetadata server-side.
- If the user wandered into a domain outside the
standard subject modes ("other" mode), suggestion
may be null and that's fine.
No commentary outside the JSON.
```
## 5. Use cases & content to include
Build dedicated UI flows or seed sessions for each of these —
they tell you what content the app must support.
- **The calculus problem at the kitchen table.** A student has
`∫ x · sin(x²) dx` on a worksheet. They prop the phone in a
stand, point the camera at the paper, tap **Start**. The model
greets, asks what they've tried. They explain. The model asks
"what would you pick as `u`?". They pick `u = x²`. The model
asks what `du` is. They work it out. The model asks how the
integral transforms. They get there in four minutes with one
small hint. Transcript saves with the photographed working.
- **The React `useEffect` infinite loop at 11pm.** A junior
developer has a `useEffect` firing every render. They point
the phone at the laptop screen showing VS Code. The model
reads the code, asks "what's in your dependency array?" The
developer realises they put the state setter in there. The
model asks "what should be in there instead?". The developer
types the fix themselves. The model never pastes a solution.
- **The windowpane test.** A baker is trying their first
sourdough. The recipe says "knead until the windowpane test
passes" and they have never heard of it. They prop the phone
on a stand over the dough, tap **Start**. The model asks what
the dough feels like, asks them to tear off a small piece and
stretch it. They do. The model asks what they see when they
hold it up to the light. They see translucency without
tears. The model says "that's the windowpane — you're
there".
- **The `#REF!` spreadsheet error.** A user has a VLOOKUP
returning `#REF!` in cell `D14`. They show the spreadsheet
to the camera. The model asks what the third argument of
the VLOOKUP is (the column index). They say "5". The model
asks how many columns the lookup table has. They count:
"4". The user fixes it themselves and the model marks
arrival.
- **The wobbly Billy bookcase.** A new-flat dweller has a
Billy bookcase assembled but wobbly. They show the camera
the back. The model asks if the cam-locks at the bottom
are tight. They check and find two loose. The model asks
what tool they'd use. They get a flathead screwdriver. The
model waits while they tighten. The bookcase stops
wobbling. Arrival.
- **The lawnmower that won't start.** A user has a
petrol lawnmower that pulls but won't fire. They show the
spark plug area, the air filter housing, the fuel tank.
The model surfaces a not-a-substitute banner the moment
it sees a small-engine carburettor (chemical_handling for
fuel, professional territory for repairs) and continues
Socratically: "when did you last change the spark plug?"
The user replaces it themselves and the mower fires.
Arrival, with the banner visible the whole time.
- **The bicycle derailleur that won't shift.** A weekend
cyclist has a rear derailleur skipping gears. They show
the cassette and the derailleur. The model asks them to
shift through the gears slowly while watching. They see
the chain hesitating between two specific cogs. The model
asks what the limit screws do. The cyclist remembers
reading about high and low limits, finds the H screw,
adjusts a quarter-turn. The chain settles. Arrival.
- **The chord voicing on guitar.** A student is learning
the F-major barre chord and can't get a clean sound.
They show their hands on the fretboard. The model asks
which strings are buzzing. They identify the B and high
E. The model asks where their index finger is pressing
on those strings. They look. The model never plays the
chord for them; it asks the next question. They reposition
their finger and get the clean sound. Arrival.
- **The Spanish menu on holiday.** A traveller in Seville
has a menu they want to understand without just
translating everything. They show the camera "merluza a
la plancha con piquillos". The model asks what they
recognise. They get "a la plancha — grilled". The model
asks what they think "merluza" might be (a fish). They
guess hake. The model confirms with "you've got it — a
white fish related to cod". They order with confidence.
- **The family-mode kid working on long division.** A
parent has switched family-mode on; their 10-year-old
is working on `742 ÷ 7`. The kid asks the model "what's
the answer?" The model calmly redirects: "let's work it
out together. What's the first thing you do in long
division?" The kid works through it. Arrival.
## 6. Page structure
Build the following screens / sections in this order. Adjust
copy to fit the voice, but keep the structural intent.
1. **Welcome / sign-in.** A photographed-looking shot of a phone
in a stand on a kitchen table at golden hour, a notebook open
showing a handwritten calculus problem, the phone showing the
Live Tutor record screen. One paragraph: "Live Tutor is a
calm voice that walks you through whatever you're stuck on.
Point your camera. Get the next question — never the answer."
Single Google sign-in button; Apple sign-in next to it.
Below: a **Hear the voice** button that plays a 12-second
pre-recorded audio sample of the tutor saying a sample
Socratic exchange. Below that: **Try a practice session** →
loads one of the demo problems from section 8a.
2. **Subject chips screen** (shown when the user taps **Start a
session** from the home screen). Seven chips: *Maths*, *Code*,
*Recipe*, *Spreadsheet*, *DIY / repair*, *Music*, *Language*.
An eighth small text "Or just point your camera — I'll figure
it out". Above the chips, the live camera preview at half-
height so the user sees the framing before they tap.
3. **Active session screen.** Full-screen camera preview. At
the top, the hint-budget dots (three filled), the session
timer, a *Pause* button, and a small *Show me what you see*
toggle. At the bottom, a live transcript pane (~1/3 height,
scrollable mid-session). Floating above: *I'm stuck*, *I
think I've got it*, *End session*. The not-a-substitute
banner, when surfaced by a safety flag, appears as a slim
pinned banner above the transcript pane; non-dismissable
for the remainder of the session.
4. **Arrival moment.** When the model calls `markArrival()`,
the screen briefly dims around a centred card: "You did it —
want to save the transcript?". The card shows the most-
recent still the model commented on, the topic-summary one-
liner, the time taken, the hint count. Two buttons: *Save
transcript* (primary), *End without saving* (secondary).
5. **Session library.** A chronological feed of past sessions.
Each card: subject icon, topic summary, duration, hint
count, the moment-of-arrival timestamp. Search bar at the
top. Filter by subject chip.
6. **Session detail view.** Full Socratic transcript on the
left half; the still frames captured during the session as
a scrollable strip on the right. Each model turn timestamped;
each hint event marked with a small dot in the margin. The
moment-of-arrival highlighted. Buttons at the top: *Export
PDF*, *Export Markdown*, *Email to myself*, *Resume this
session* (only if the previous session ended at the 2-min
cap without arrival).
7. **Practice session library.** A small built-in library of
starter problems per subject. Each problem is a card with
the problem statement, the subject chip, and a *Try this
session* button that pre-loads a demo camera view (using a
cached sample image) and a curated `SessionPreamble`.
8. **Family-mode panel.** Settings → Family-mode. Toggle.
Strict-subject-scope toggle (defaults on). Hint budget per
subject. Redirection style: gentle / firmer. Parent digest
email frequency: never / weekly / never (default never).
9. **Settings & privacy.** Voice mode (default / calm-acoustic).
Camera preview position (top / bottom). Default subject
mode (off). Workspace integration toggle (post-I/O; off by
default). "Delete my account" / "Delete a session forever"
with a 60-second cool-off. Privacy panel restates the not-
trained-on policy in plain English.
10. **Footer.** "Made for the moment you're stuck." Privacy:
"Your sessions are yours. We never train on them."
Capabilities `(i)` icon in header.
## 6b. First-visit onboarding
Show a **first-visit onboarding** the first time a visitor
lands on the app (detect via `localStorage` flag; do not show
on return visits). Three slides, dismissible at any time.
Persistent re-entry: a `?` icon in the header reopens it.
**Slide 1 — What this is.**
- Headline: "Welcome to Live Tutor."
- Subhead: "Point your camera at what you're stuck on. Get a
calm voice that asks the next question — never gives the
answer."
- One paragraph (≤ 60 words): "Live Tutor uses Gemini 3.1
Flash Live to see what your camera sees and hear what you
say, in real time. It's Socratic by design — it asks what
you've tried, prompts the next step, waits while you work,
and celebrates when you arrive. Three hints per session,
visible at the top."
- Visual: an annotated illustration of the active session
screen — hint-budget dots labelled, camera preview
labelled, transcript pane labelled. Not a generic phone
mockup.
**Slide 2 — Try it now.**
- Headline: "Hear the voice, try a practice problem."
- A **Hear the voice** button — plays a 12-second pre-
recorded sample of the tutor voice doing a Socratic
exchange. (Cached MP3 generated via
`gemini-3.1-flash-tts-preview`; not a live session.)
- A **Try a practice problem** button — opens a session
pre-loaded with one of the demo problems from section 8a
(a calculus problem, a React-hook bug, a sourdough step,
a Billy bookcase, or a Spanish menu).
**Slide 3 — How to remix this.**
- Headline: "Make this yours."
- Three short bullets:
- "Swap the practice problem library in `/data/practice/`
for your own seed problems."
- "Adjust the per-subject preambles in
`/server/preambles/` to fit your tutoring style and
vocabulary."
- "Wire up your Gemini API key and Firebase project via
the env-var list in the capabilities panel."
- Primary CTA: "Use this template" → links to AI Studio
Build remix entry point.
- Secondary: "Just exploring — close" (sets
localStorage flag, never auto-shows again).
**Accessibility:** focus trap, `Esc` closes, `role="dialog"`,
`aria-modal="true"`, `aria-labelledby`, focus restored to
trigger on close. Respect `prefers-reduced-motion`.
**Don't:**
- Don't gate content behind the modal. The page beneath
must be fully usable.
- Don't auto-reshow on return visits.
- Don't include unrelated CTAs.
- Don't ship without the **Hear the voice** sample — the
voice IS the product, and visitors should hear it before
they commit a Live API session.
## 6c. Capabilities info button (persistent in header)
Add a persistent `(i)` icon in the top-right of the header.
Click → opens a modal/panel titled **"What powers this app"**.
**Panel contents (in this order):**
**Gemini capabilities used (the hero list):**
- **Gemini 3.1 Flash Live (`gemini-3.1-flash-live-preview`)** —
the live audio + video model that sees what your camera sees
and speaks back in real time. Post-Google-I/O 2026 acoustic
nuance means pitch and pace sound considered, not robotic.
Audio + video sessions cap at 2 minutes per turn — the app
resumes seamlessly when you cross that limit.
- **Gemini 3.5 Flash** — the post-I/O default flagship.
Powers the subject classifier (figures out whether you're
doing maths, code, recipe, etc.), the session preamble
assembly (sets up the right vocabulary for the live
session), the safety classifier (flags
electrical/gas/structural work), and the transcript post-
processor (cleans the saved record).
- **Gemini 3.5 Flash + grounded search** — suggests one
follow-up problem at the end of each successful session,
grounded in real textbook / online sources (citations
surfaced).
- **Nano Banana 2 (`gemini-3.1-flash-image`)** — generates
the welcome and empty-state imagery (a phone in a stand
on a kitchen table at golden hour with a notebook).
- **Gemini 3.1 Flash TTS (`gemini-3.1-flash-tts-preview`)**
— used once during build to record the 12-second voice
sample on the welcome screen. Cached as a static MP3;
not called per visit.
- **Live API function calling** — three synchronous
functions (`markHint`, `markArrival`, `requestStillFrame`)
let the model coordinate with the client without breaking
the conversational flow.
- **Firebase Auth** — Google and Apple sign-in.
- **Firestore** — stores your sessions, syncs across
devices.
- **Firebase Storage** — keeps the still frames captured
during sessions.
- **Cloud Run gateway** — a server function proxies the
Live API websocket so your `GEMINI_API_KEY` never
leaves the server. Post-I/O 2026 you get free deploy of
the first 2 apps to Cloud Run.
**The Socratic discipline (the rule that defines this app):**
- The model never gives the answer outright. It asks the
next question. Three hints per session, visible at the top
as filled dots. When the budget is exhausted, the model
defaults to encouragement and reflection rather than
capitulating.
**The safety contract:**
- The still-frame safety classifier runs on every captured
still. If it sees mains electrical, gas, load-bearing
structural, medical wound, firearm, chemical handling, or
child-safety territory, the app surfaces a not-a-
substitute banner that stays visible for the rest of the
session. The Live API model is instructed never to
authorise such work — only to help the user think through
what they see and what a qualified professional would
check.
**Backend services this app depends on:**
- Auth: see section 4b
- Database: see section 4b
- Storage: see section 4b — REQUIRES manual enable in
Firebase console; AIS Build does not auto-provision
Storage today.
- Email: see section 4b — magic-link transcript sharing
requires the sender domain to be authorised in Firebase
Auth.
- Apple sign-in: optional, requires an Apple Developer
account and Service-ID config. See section 4b.
- Payments: see section 4b (not used in v1).
- External APIs: see section 4b.
**Environment variables you'll need to configure:**
- `GEMINI_API_KEY` — your Google AI Studio API key
- `FIREBASE_PROJECT_ID` — your Firebase project id
- `FIREBASE_SERVICE_ACCOUNT` — service-account JSON
(server-side only)
**Cost + privacy notes:**
- One short paragraph per cost-sensitive capability: Live
API sessions are billed by streaming-second; a 4-minute
session (2 segments) is approximately $0.18. The text
calls (classifier + preamble + post-processor + safety)
add ~$0.01/session.
- One short paragraph on privacy: where the data lives
(your Firebase project), how to delete it (Settings →
"Delete this session forever" — gone in 60 seconds), what
is never sent for training, what is never stored (raw
video frames beyond the live session window).
**Documentation links:**
- AI Studio Build docs
- Gemini API Live (`gemini-3.1-flash-live-preview`) docs
- Gemini API multimodal image, structured output, grounded
search docs
- Firebase Auth, Firestore, Firebase Storage docs
**Accessibility:** same standards as the onboarding modal —
focus trap, `Esc`, ARIA, restored focus.
**Behaviour:**
- Always available — single click from anywhere in the app.
- Tooltip on the `(i)` icon: "How this app is built".
- Mobile: opens as a full-screen sheet that slides up.
- Should be the most honest part of the app — never hand-
wave service requirements; never say "AI" without naming
the specific Gemini model and capability.
## 6d. Detailed cost breakdown (deployer reads this BEFORE shipping)
- **Live API session (`gemini-3.1-flash-live-preview`)** —
billed by streaming-second. As of 2026-06, ~$0.075/min of
audio in, ~$0.30/min of video in, ~$0.30/min of audio
out (approximate; check current pricing). A typical 4-min
session (one resume) ≈ ~$0.18. A 2-min session (no
resume) ≈ ~$0.09. **Plan for ~$0.10–$0.20 per session.**
- **Subject classifier (Gemini 3.5 Flash, low thinking)** —
~10 frames + a short system instruction, ~3,000 input
tokens and ~150 output tokens → ~$0.006 per session.
- **Session preamble assembly (Gemini 3.5 Flash, low)** —
~500 input tokens + 300 output → ~$0.004 per session.
- **Safety classifier (Gemini 3.5 Flash, low)** — runs on
each `requestStillFrame()` capture. Typical session has
2–4 stills → ~$0.004 per still → ~$0.012 per session.
- **Transcript post-processor (Gemini 3.5 Flash, low)** —
~8,000 input tokens (transcript + still descriptions) +
1,000 output → ~$0.02 per session.
- **Next-session suggestion (Gemini 3.5 Flash + grounded)** —
~$0.005 per call, run once per successful session.
- **Hero / empty-state image (Nano Banana 2)** — ~$0.03/image.
Generated once per app install (cached client-side).
- **TTS voice sample (Gemini 3.1 Flash TTS)** — recorded
ONCE at build time, cached as static MP3 in
`/public/audio-samples/welcome-12s.mp3`. Approximately
$0.0015 to generate; zero cost per visitor.
- **Expected per-session cost (typical):** ~$0.13–$0.23
Gemini API spend.
- **Heavy user (1 session per day):** ~$5/month Gemini
total.
- **Still-frame storage:** Firebase Storage standard tier
~$0.026/GB/month. A 1080p JPEG ≈ ~400 KB; 50 stills/month
≈ ~20 MB ≈ ~$0.0005/month.
- **Cloud Run gateway:** free tier covers the first 2
million requests/month and 360k vCPU-seconds. Live API
proxying uses minimal CPU because frames pass through.
Post-I/O 2026 free 2-app Cloud Run deploy covers this
app out of the box.
## 7. Design language
- **Mood:** A calm desk lamp on a study table at 11pm. The
phone in a stand. A notebook open. A laptop screen
showing code. A kitchen counter with a dough at the
windowpane test. The carburettor housing open on a
workshop bench. Not a tech-product showcase. Not a flashy
AI demo. The moment between "I'm stuck" and "oh — I see".
- **Typography:** Clean grotesque for app chrome and the
live transcript (Inter or Geist). Display serif for
session titles and the welcome headline (Source Serif Pro
or Fraunces). A monospaced face for code-mode transcripts
(JetBrains Mono or SF Mono). Body 16-17px, generous
line-height in the transcript (1.7+) so the live caption
is comfortable to read mid-session.
- **Palette:** Warm parchment background `#F5F1EA` for the
notebook surfaces and session library cards. Deep ink
`#1B1714` for body. Tutor violet `#7C3AED` for the hint-
budget dots, the active record button, and the moment-of-
arrival highlight (matches the user's preferred Apple-
doc accent). A soft tutor-violet tint `#EDE9FE` for the
hint-budget background and the live-transcript pane.
Tutor-pink `#DB2777` accent (sparingly, for the *I think
I've got it* button to differentiate from the regular hint
request). Highlight yellow `#CA8A04` reserved for the
not-a-substitute safety banner with soft tint `#FEF9C3`.
Tutor-green `#16A34A` only for the arrival confirmation
dot.
- **Imagery:** Photographic. A phone in a stand on a
kitchen table at golden hour. A laptop screen showing
code with a window-light reflection. A wooden workbench
with the bicycle frame upside-down. Generated via Nano
Banana 2 with prompts emphasising warm light, real
imperfection (paper grain, fingerprint smudge on the
phone screen), no people, no commercial branding.
- **Hand-feel touches:** The hint-budget dots fade rather
than vanish when used — the previous filled state is
ghosted for half a second so the user sees what they
spent. The live transcript pane has a soft scroll-shadow
at the top and bottom so it reads like a continuous strip
of paper passing through. The moment-of-arrival card
has a thin watercolour-wash background, suggesting
something hand-marked. The record button breathes (a
4-second sinusoidal opacity oscillation between 1.0 and
0.92) when the Live API session is connecting, settling
to steady on connection.
- **Spacing:** consistent 4-px base. Generous whitespace —
the live transcript and the camera preview need air.
- **Radius:** consistent token set (e.g. 8 / 14 / 22 px).
Transcript turn cards use 8; the arrival card uses 14;
the welcome card uses 22.
- **Shadows:** subtle, warm-tinted. Avoid heavy drop-
shadows. The hint-budget dots have no shadow.
- **Motion:** purposeful — the record button's breathing
oscillation while connecting; the hint dot's fade-on-
spend; the moment-of-arrival's brief dim-around. Respect
`prefers-reduced-motion`: replace the oscillation with a
static colour change; replace the fade with an instant
state change; replace the dim with a thin highlight.
- **States:** every interactive element has hover, focus,
active, disabled. Loading uses skeletons (the camera
preview's "connecting…" placeholder is a soft pulse on a
parchment rectangle, not a spinner). Empty states have
domain-specific next-action labels ("Try the calculus
practice problem", "Try the React-hook practice
problem").
## 8. Content generation rules
- Write **realistic, specific copy**. NO Lorem Ipsum. NO
generic placeholders.
- Invent plausible names, problem statements, transcript
snippets, and arrival-summaries that fit the domain. When
inventing, lean on realistic patterns — `∫ x · sin(x²)
dx`, a `useEffect` with a missing dependency, a windowpane
test on sourdough, a VLOOKUP with the wrong column index,
a wobbly Billy bookcase with loose cam-locks.
- Tone: warm, direct, calm. This template is for someone
who is stuck — they do not need cheerleading and they do
not need to feel small.
- Headlines: punchy and concrete. No 'Empower your X'
filler. No 'AI-powered'. No 'Revolutionize your learning'.
- Body copy: short paragraphs (2-4 sentences). Use lists
where appropriate.
- Plain language. Avoid jargon — except where the user
already speaks it (the maths student wants to see
"integral", "substitution"; the developer wants to see
"useEffect", "dependency array"; the baker wants to see
"windowpane test", "autolyse").
- Where the app outputs AI-generated content, never label
it as "AI says" — let the tutor voice speak naturally.
- The not-a-substitute disclaimer copy is the SAME wording
every time, surfaced as a banner: "This isn't a substitute
for an electrician / plumber / clinician / structural
engineer. I can help you think through what you're
seeing, but I'm not qualified to authorise the work."
(Verb adjusts to the safety category.)
## 8a. Seed content (use these specific examples)
Anchor every generated copy + sample data point in the
content below.
**Sample practice problems (built-in library):**
- **Maths — Calculus by substitution.** Problem statement
shown on screen: `∫ x · sin(x²) dx`. Pre-loaded camera
view: a handwritten version on lined paper.
`SessionPreamble`:
- vocabulary_anchors: ["integral", "substitution",
"u-substitution", "antiderivative", "differential",
"du", "limits of integration"]
- socratic_patterns: ["What would you pick as u?", "What
does du equal?", "How does the integral transform?",
"What does the new integral evaluate to?"]
- safety_watchlist: []
- initial_greeting_one_line: "I can see your integral —
walk me through what you've already tried."
- **Code — React useEffect infinite loop.** Problem
statement: a `useEffect` hook that includes a setter
function in its dependency array, causing infinite re-
renders. Pre-loaded camera view: a screenshot of VS Code
with the problematic hook visible. `SessionPreamble`:
- vocabulary_anchors: ["useEffect", "dependency array",
"infinite loop", "stale closure", "render cycle",
"state setter", "mount", "unmount"]
- socratic_patterns: ["What's in your dependency array?",
"What does each value in there control?", "What
triggers a re-render here?", "What would you want
that array to contain instead?"]
- safety_watchlist: []
- initial_greeting_one_line: "I can see your useEffect
in VS Code — walk me through what you're trying to
do here."
- **Recipe — Sourdough windowpane test.** Problem
statement: "knead until the dough passes the windowpane
test". Pre-loaded camera view: a ball of dough on a
floured wooden counter. `SessionPreamble`:
- vocabulary_anchors: ["windowpane test", "gluten",
"autolyse", "bulk ferment", "stretch and fold",
"elasticity", "translucent", "tear"]
- socratic_patterns: ["What does the dough feel like
right now?", "How does it stretch?", "What do you
see when you hold it up to the light?", "Does it
tear, or does it stretch thin?"]
- safety_watchlist: []
- initial_greeting_one_line: "I can see your sourdough
in progress — what does the dough feel like right
now?"
- **Spreadsheet — VLOOKUP #REF! error.** Problem statement:
cell `D14` shows `#REF!` from a VLOOKUP formula. Pre-
loaded camera view: a screenshot of a Google Sheet with
the formula bar showing
`=VLOOKUP(A14, $B$2:$E$50, 5, FALSE)`. `SessionPreamble`:
- vocabulary_anchors: ["VLOOKUP", "lookup table",
"column index", "exact match", "approximate match",
"#REF!", "#N/A", "absolute reference"]
- socratic_patterns: ["What's the third argument of your
VLOOKUP?", "How many columns does your lookup table
have?", "What range did you give VLOOKUP to search?"]
- safety_watchlist: []
- initial_greeting_one_line: "I can see your #REF! in
D14 — what does your VLOOKUP formula look like?"
- **DIY — Wobbly Billy bookcase.** Problem statement: a
Billy bookcase assembled but wobbly. Pre-loaded camera
view: a photo of the back of a bookcase with cam-locks
visible. `SessionPreamble`:
- vocabulary_anchors: ["cam-lock", "dowel", "wooden peg",
"Allen key", "flathead", "Phillips", "back panel",
"carcass"]
- socratic_patterns: ["Where do you feel the wobble?",
"Are the cam-locks at the bottom snug?", "Is the back
panel fully seated in its groove?"]
- safety_watchlist: ["load_bearing"]
- initial_greeting_one_line: "I can see your Billy
bookcase — where do you feel the wobble?"
- **DIY — Lawnmower won't start.** Problem statement: a
petrol lawnmower pulls but won't fire. Pre-loaded camera
view: a photo of a lawnmower with the air filter housing
open. `SessionPreamble`:
- vocabulary_anchors: ["spark plug", "air filter",
"carburettor", "choke", "primer bulb", "fuel line",
"compression", "flywheel key"]
- socratic_patterns: ["When did you last change the
spark plug?", "What does the air filter look like?",
"Is fuel reaching the carburettor?", "What did the
spark plug look like when you pulled it?"]
- safety_watchlist: ["chemical_handling",
"other_professional"]
- initial_greeting_one_line: "I can see your mower — and
I'll flag a heads-up that fuel and small engines are
professional territory. When did you last change the
spark plug?"
- **Music — F-major barre chord buzzing.** Problem
statement: an F-major barre chord on guitar with buzzing
on the B and high E strings. Pre-loaded camera view: a
photo of a guitar fretboard with a hand attempting the F
chord. `SessionPreamble`:
- vocabulary_anchors: ["barre", "fret", "fingerboard",
"string", "buzz", "thumb position", "index finger",
"knuckle"]
- socratic_patterns: ["Which strings are buzzing?",
"Where is your index finger pressing on those
strings?", "What does your thumb look like behind
the neck?", "How close to the fret is your
pressure?"]
- safety_watchlist: []
- initial_greeting_one_line: "I can see your F-chord
attempt — which strings are buzzing?"
- **Language — Spanish menu in Seville.** Problem
statement: a menu item reads "merluza a la plancha con
piquillos" and the user wants to understand it without
pure translation. Pre-loaded camera view: a photo of a
menu page. `SessionPreamble`:
- vocabulary_anchors: ["merluza", "a la plancha",
"piquillos", "white fish", "grilled", "Spanish
cuisine", "cognate"]
- socratic_patterns: ["What do you recognise here?",
"Does 'plancha' remind you of any English word?",
"What kind of food might 'merluza' be from the
section it's in?"]
- safety_watchlist: []
- initial_greeting_one_line: "I can see the menu —
what do you already recognise?"
**Sample arrival transcripts (the moment-of-arrival lines
the model should produce):**
- Calculus arrival: "That's it. You did the substitution; I
just kept you company. Save the transcript?"
- React arrival: "Nice catch on the dependency array. Type
the empty brackets and watch your component settle."
- Sourdough arrival: "That's the windowpane — you're there.
The dough's ready to shape."
- VLOOKUP arrival: "Four columns, asked for the fifth. You
saw it. Change the five to a four and it'll resolve."
- Billy bookcase arrival: "Two snug cam-locks, no more
wobble. Want me to save the transcript?"
- Lawnmower arrival: "Fresh plug, fresh start. Remember the
banner — for anything fuel-system or carburettor-internal,
a mechanic is the call."
- F-chord arrival: "That's a clean F. Your index finger
finally got the rolling pressure right behind the second
fret."
- Spanish menu arrival: "You've got it — grilled hake with
piquillo peppers. Order with confidence."
**Sample voice copy (UI strings):**
- Welcome subhead: "Point your camera at what you're stuck
on. Get a calm voice that asks the next question — never
gives the answer."
- Hear-the-voice button: "Hear the voice"
- Practice-session CTA: "Try a practice problem"
- Subject chips header: "What are you working on?"
- Hint-budget tooltip: "Three small hints per session. I'll
use them sparingly."
- I'm-stuck button: "I'm stuck"
- I-think-I've-got-it button: "I think I've got it"
- End-session button: "End session"
- Connecting state: "Opening the line — give me a second."
- Reconnecting after 2-min cap: "Picking up where we left
off."
- Arrival card heading: "You did it."
- Arrival card subhead: "Want to save the transcript?"
- Save-transcript button: "Save transcript"
- Not-a-substitute banner (electrical): "This isn't a
substitute for an electrician. I can help you think
through what you're seeing, but I'm not qualified to
authorise the work."
- Not-a-substitute banner (medical): "This isn't a
substitute for a clinician. I can help you describe what
you see, but I'm not qualified to diagnose or treat."
- Low-confidence framing prompt: "Camera's a little dark —
can you move closer or add light?"
- 2-min warning: "We've got about 15 seconds left on this
segment — I'll pick up right where we are."
- Resume prompt: "Last time we got as far as picking u =
x². Want to keep going from there?"
- Family-mode redirect: "Let's get back to the problem on
your paper."
**Sample seed transcript turn-by-turn (for the demo session
detail view):**
- 0:00 — model: "I can see your integral — walk me through
what you've already tried."
- 0:08 — user: "I tried integration by parts but it got
messy. I think substitution might work."
- 0:14 — model: "Good instinct. What would you pick as u?"
- 0:22 — user: "Probably x squared, since its derivative
is x and there's an x in front."
- 0:30 — model: "That's the choice I'd make. What's du?"
- 0:36 — user: "du equals 2x dx."
- 0:41 — model: "You've got 2x dx on the right and x dx
in the integral — what does the integral become in
terms of u and du, once you handle that factor of 2?"
- 1:02 — user: "Half… half integral of sin u du?"
- 1:08 — model: "That's it. Now finish."
- 1:24 — user: "Negative one-half cosine u plus C, which
is negative one-half cosine x squared plus C."
- 1:32 — model: "Hold it up for me?"
- 1:36 — user: [holds paper to camera] — model called
requestStillFrame() at 1:37
- 1:42 — model: "That's right. You did the substitution; I
just kept you company. Want me to save the transcript?"
- 1:46 — model called markArrival("user solved
∫ x · sin(x²) dx by u-substitution, u = x², arrived at
-1/2 cos(x²) + C", 0.95)
## 9. Media & assets
- **Hero image (landing screen):** A photographed-looking
shot of a phone in a stand on a kitchen table at golden
hour, a notebook open showing a handwritten calculus
problem, the phone screen showing the Live Tutor record
view. Generate via Nano Banana 2 with a prompt
emphasising "warm afternoon light through a kitchen
window, no people in frame, a phone in a wooden stand
showing a clean app interface, a paper notebook open
beside it with a handwritten integral, soft shadow under
the phone, real worn paper, a coffee mug just out of the
frame".
- **App icon / wordmark:** Set in the display serif.
Slightly worn paper texture behind it. No icon — just
type ("Live Tutor").
- **Empty-state illustration:** A simple line drawing of a
phone in a stand pointing at a notebook with a question
mark on the page. Hand-drawn aesthetic, not a flat icon.
Generate once at build time via Nano Banana 2
(`gemini-3.1-flash-image`), 1:1 WebP at 1024×1024, prompt:
"single hand-drawn ink line illustration of a smartphone in
a small upright stand on the left tilted slightly toward an
open paper notebook on the right with a single hand-drawn
question mark on the visible page, off-white paper
background, slight pen imperfection, no shading, no colour
fill, no text, no commercial branding". Ship as a seed asset
at `/public/samples/empty-state-phone-notebook.webp`.
- **Practice-problem cover images:** One photographed-
looking image per practice problem, generated via Nano
Banana 2. The maths one shows the handwritten integral on
paper; the code one shows a laptop screen with VS Code;
the recipe one shows dough on a wooden counter; the
bookcase one shows the back of a half-assembled Billy;
etc.
- **Subject-chip icons:** Hand-drawn icons (lucide-react
with hand-tuned strokes) — a square root sign for maths,
an angle bracket for code, a whisk for recipe, a small
grid for spreadsheet, a hex key for DIY, a treble clef
for music, a speech bubble for language.
- **Cached voice sample audio:** Generated once via
`gemini-3.1-flash-tts-preview`, served as a static MP3
from `/public/audio-samples/welcome-12s.mp3`. The sample
is a 12-second Socratic exchange: model asks "What would
you pick as u?", a beat of silence, user voice (also
TTS) says "x squared", model says "That's the choice I'd
make. What's du?".
- **Stock fallbacks:** If Nano Banana image generation
fails, fall back to the photographed sample image from
`/public/samples/sample-desk.jpg` (3:2 WebP, 2048×1365 —
ship as a seed asset; recreate via Nano Banana 2
(`gemini-3.1-flash-image`) with the prompt: "photographic
phone in a wooden stand on a kitchen table at golden hour,
paper notebook open beside it with a handwritten integral
on the visible page, soft shadow under the phone, real worn
paper, a coffee mug just out of the frame, no people, no
commercial branding"). Never to a generic AI-generated
abstract.
- **Generated imagery:** prefer Nano Banana 2 over stock
photography. Prompt for warmth, asymmetry, slight
imperfection. Avoid glossy AI render look.
- **Optimisation:** WebP/AVIF, `loading="lazy"`, explicit
`width`/`height` to prevent layout shift.
- **Icons:** `lucide-react` for UI. Use sparingly.
## 10. Interactivity & states
- Every interactive element has hover, focus, active, and
disabled states.
- Forms validate inline and show specific error messages.
"Camera permission denied — check your browser's site
settings" is the right shape, not "Error".
- Loading states use skeletons that match the eventual
layout. The connecting state of the Live API session
uses the breathing record button (a 4-second sinusoidal
opacity oscillation), not a spinner.
- Empty states explain the next action with domain-
specific labels: "Try the calculus practice problem",
"Try the React-hook practice problem". Never a generic
"Get started".
- Smooth scroll for in-page anchors.
- The Live API session's transcript streams in real time —
caption appears as the model speaks, slightly behind the
audio. User turns appear when audio transcription is
ready.
- If the Live API connection fails, show a calm, specific
error ("We couldn't open the live session — check your
network and try again. Your session has not been
charged.") and offer retry.
- The hint-budget dots animate as they're spent: the dot
fades from filled to outline over 300ms, with a faint
ghost of the filled state lingering for 500ms. With
`prefers-reduced-motion`: instant state change.
- The not-a-substitute banner, when surfaced, slides in
from the top of the transcript pane over 250ms and stays
pinned for the rest of the session. With `prefers-
reduced-motion`: appears instantly.
- The arrival moment dims the screen around a centred card
over 400ms; the card fades in over 200ms. With `prefers-
reduced-motion`: a thin highlight ring around the card
instead of the dim.
- All AI-generated content respects token-by-token
streaming where the Live API provides it; the caption
pane updates as audio arrives.
- The 2-minute video session warning surfaces at 1:45 as
a slim non-blocking inline note ("about 15 seconds left
on this segment — I'll pick up right where we are").
- Resume of a previously-capped session is one tap from
the session library; the resume opens a new Live API
connection with a fresh system message summarising where
the previous segment left off (text-only summary, not
a re-stream of frames).
## 11. Tech & responsive requirements
- **Stack:** React + TypeScript + Tailwind CSS. Functional
components + hooks. Use Shadcn UI primitives where
appropriate.
- **Build runtime:** AI Studio Build — full-stack with
Cloud Run server-side functions. All Gemini API calls
happen server-side; API key lives in Secrets Manager,
never in client bundle.
- **Live API gateway:** a Cloud Run server function opens
the Live API websocket using the project's
`GEMINI_API_KEY` and proxies frames + audio between the
client and Google. The client receives short-lived
ephemeral tokens (10-min TTL) bound to the user's
Firebase Auth UID. **Do not** put `GEMINI_API_KEY` in
the client bundle.
- **Audio capture:** the Web Audio API (specifically
`AudioWorklet`) for capture + downsample to 16kHz PCM
mono. `getUserMedia` for the audio + video track. The
client must request `audio` and `video` permissions
separately and degrade gracefully if only audio is
granted (audio-only mode allows 15-minute sessions).
- **Audio playback:** Web Audio API for playback of the
PCM 24kHz mono stream from the Live API; queueing must
handle jitter without dropping audio (a 200ms playback
buffer is a reasonable default).
- **Video capture:** `getUserMedia` with `video: { width:
1280, height: 720 }` constraint; sample at 1 FPS to the
Live API channel via `requestAnimationFrame`-throttled
capture.
- **Still-frame capture (non-blocking ring buffer):** the
`requestStillFrame()` function call MUST NOT block the Live
audio channel. The client maintains a 3-frame 1080p ring
buffer continuously sampled from the video stream. When the
Live API fires `requestStillFrame`, the client dispatches the
latest cached frame asynchronously to an upload worker; the
active audio/voice loop is never blocked, and the upload
happens off the bidirectional gRPC/WebSocket stream. The
worker uploads to Firebase Storage, server-side re-uploads to
the Gemini Developer API Files API, and surfaces the resulting
`files/*` resource name (passed via `fileData.fileUri`) back
to the gateway so the model can reference it in the next turn.
- **Live API session state recovery.** The Live session caps at
2 minutes (audio+video) / 15 minutes (audio-only); on every
Live tick the client persists a `SessionSyncState` to
`sessionStorage`, and on reconnect it passes a concise context
summary block into the first system message of the next
handshake cycle so the model picks up where it left off:
```typescript
interface SessionSyncState {
activeSessionId: string;
accumulatedSegments: Array<{ speaker: string; text: string; timestamp: number }>;
currentActiveRubricIndex: number; // current step in the Socratic plan
activeStillUris: string[]; // files/* refs captured so far
hintEventsSoFar: Array<{ index: number; size: 'small' | 'medium'; givenAtIso: string }>;
}
```
- **Model selection:** explicitly pin
`gemini-3.1-flash-live-preview` for the live session and
`gemini-3.5-flash` for every text-only call. Set
`thinkingLevel: "low"` on every 3.5 Flash call. Do NOT
set `thinkingConfig` on the Live API, image-gen, or TTS
calls.
- **Database:** Firestore (auto-provisioned by AI Studio
Build). Show the practice library on first launch.
- **Auth:** Firebase Auth — Google sign-in by default;
Apple sign-in next to it; magic-link email as fallback.
- **Storage:** Firebase Storage for still frames. Pre-
signed URLs only.
- **Mobile-first.** Verify layouts at 375 px (iPhone SE),
768 px (iPad), 1024 px, 1440 px+. The active session
screen MUST work at 375 px with the camera preview, the
hint dots, the transcript pane, and the action buttons
all visible without scrolling.
- Use `clamp()` for fluid typography. Prefer container
queries over media queries for component-level
responsiveness.
- Use `dvh` / `svh` instead of `vh`. Respect safe-area
insets on iOS (the floating action buttons must clear
the home indicator).
- Zero horizontal overflow at any width. Zero layout
shift on load.
- Persist user data in Firestore. The active session is
ephemeral until the user taps **Save transcript** —
saved transcripts include the turn list, the still URIs,
the hint events, the arrival event.
- Wake-lock during an active session via the Screen Wake
Lock API (the user should not have their screen sleep
mid-Socratic-question); falls back gracefully if
unsupported.
## 12. Accessibility (WCAG 2.2 AA)
- Semantic HTML — `header`, `nav`, `main`, `section`,
`article`, `footer`.
- All interactive controls reachable by keyboard with a
visible focus ring.
- Color contrast ≥ 4.5:1 for body, 3:1 for large text and
UI components. Tutor-violet `#7C3AED` on parchment
`#F5F1EA` is verified ≥ 4.5:1.
- All images have meaningful `alt` text. The practice-
problem cover images use descriptive alt ("photograph
of a handwritten integral ∫ x · sin(x²) dx on a paper
notebook").
- Form fields have associated `