# MUST OBEY — Mobile-first build requirements This app's PRIMARY surface is a mobile phone. Build it impeccably on mobile FIRST, then verify on tablet and desktop. Treat the rules below as non-negotiable hard constraints, not suggestions. ## Viewports to verify (every screen, every state) - 320 px, 360 px, 375 px, 390 px, 414 px, 480 px - 768 px, 834 px (iPad portrait / Pro 11) - 1024 px, 1280 px, 1440 px, 1920 px, 2560 px - Plus: 200% browser zoom, landscape orientation on every mobile width, iPhone with safe-area insets visible ## Hard layout rules - Mobile-first CSS. Default styles target mobile; `@media (min-width: ...)` for larger viewports. - Use `dvh` and `svh` instead of `vh` for full-height surfaces (iOS Safari URL-bar bug). - Use `clamp()` for fluid typography across all viewports. - Prefer container queries (`@container`) over media queries for component-level responsiveness. - Use `min(100%, ...)` widths so content never overflows. Zero horizontal overflow at any viewport. - Add `` to every page. - Apply `padding: max(safe-area-inset-X, fallback)` on every edge-bleeding container so notched iPhones in landscape never clip content. - Wide tables and code blocks scroll INSIDE their container (`overflow-x: auto`), never push the body. - Use `background-attachment: scroll` on mobile, not `fixed` (iOS Safari repaint bug). - Avoid `backdrop-filter` on animated elements. Use it sparingly on static surfaces only. - **Canvas Scaling**: Canvases must dynamically scale with window resize events and properly handle high-DPI screens (`window.devicePixelRatio`). Set physical dimensions (`canvas.width`/`canvas.height`) using pixel ratio and render relative to this grid, using CSS to control responsive viewport scaling. - **Robust Storage**: Every access to `localStorage`/`sessionStorage` (especially `JSON.parse` of loaded state or writes) MUST be wrapped in a `try-catch` block to handle disabled storage, private browsing mode, quota limits, or corrupted JSON gracefully. Fall back to a robust in-memory object store. ## Touch & accessibility - Tap targets ≥ 44 × 44 px on touch (Apple HIG). Increase to 48 px under `@media (hover: none) and (pointer: coarse)`. - All interactive controls reachable by keyboard with a visible focus ring; respect `:focus-visible`. - Color contrast ≥ 4.5:1 for body text, 3:1 for UI components. - All images have meaningful `alt`. Decorative images use `alt=""`. - Respect `prefers-reduced-motion: reduce` — zero animation durations under that query. - Forms validate inline; error messages are specific, not "Invalid input". - Modals: focus trap, `Esc` closes, `role="dialog"`, `aria-modal="true"`, focus restored on close. ## Performance bar (Lighthouse mobile, throttled 3G/4G) - LCP < 2.5 s · INP < 200 ms · CLS < 0.1 - JS bundle gzip < 200 KB mobile-first; lazy-load non-critical screens via `React.lazy` / dynamic imports. - No render-blocking resources above the fold. - Images: WebP/AVIF preferred, `loading="lazy"`, explicit `width`/`height` attributes (zero CLS), `srcset` for retina. - Videos: `preload="metadata"`, low-resolution poster, max 720p mobile fallback. Never autoplay with audio. - Fonts: `font-display: swap`; preload only the one used above the fold. - Smooth scroll honoured via CSS `scroll-behavior: smooth` with reduced-motion fallback. ## Pre-ship mobile checklist (the deployer MUST verify before declaring done) 1. Open at 375 px in DevTools — every screen scrolls vertically only; zero horizontal scroll. 2. Browser zoom 200% — layout reflows without overlap. 3. iPhone Safari with the URL bar visible AND landscape — no content under the home indicator; no notch clipping. 4. iPad portrait (768 px) and landscape (1024 px) — no awkward gaps; tablet-specific breakpoints land cleanly. 5. Tap every interactive element with a thumb at real-device size — every target is easy to hit. 6. `prefers-reduced-motion: reduce` — every transition / animation skips cleanly, scroll-behavior becomes instant. 7. Lighthouse mobile score ≥ 90 across all 4 categories. 8. Zero `console.error` and zero CLS shift in real-device testing on a mid-tier Android (e.g. Pixel 6a) and an iPhone SE. --- The original template starts below. All rules above apply on TOP of whatever this template specifies. --- # Live Tutor ## 1. Project **Live Tutor** is a calm voice that walks you through whatever you are stuck on — homework, a piece of code that won't compile, a recipe step you have never tried, an Excel formula returning `#REF!`, an IKEA instruction with the diagram missing a step, the lawnmower that won't start — by watching what your camera sees and asking you Socratic questions until you arrive at the answer yourself. It does not give you the answer. It asks what you have already tried, prompts the next move, waits while you do it, and celebrates when you get there. Drop the phone in a stand, tap **Start a session**, point the camera at the problem on the table — a calculus exercise on paper, a `useEffect` infinite loop on your laptop screen, a stuck carburettor float bowl with screws half-out, a wobbly Billy bookcase with cam-locks loose — and a Live API session opens. The model SEES what the camera sees and ASKS what you have already tried. You explain in twenty seconds. It asks the next Socratic question, anchored to what is actually on screen. You answer. Conversation continues. At the end the session is saved as a transcript with the photographed work, the steps you took, and the moment the model heard you arrive. This is the app a high-school student opens at the kitchen table when their parent has gone to bed and the calculus problem on the worksheet still has not given up its u-substitution. It is the app a junior developer opens at 11pm when a React hook is firing in an infinite loop and the Stack Overflow rabbit-hole has dead-ended. It is the app a new-flat dweller opens with the IKEA Billy bookcase sprawled across the living-room floor and a leftover cam-lock that the diagram does not show. It is the app a grandparent opens when the inherited bread recipe says "knead until the windowpane test" and they have never heard of the windowpane test. Same Socratic shape, very different shaped problems. **The 30-second demo that proves the magic:** the user puts the phone in a stand, taps **Start**, points the camera at a handwritten calculus problem — `∫ x · sin(x²) dx`. The model's voice (calm, unhurried, with the post-I/O 2026 acoustic nuance): *"Walk me through what you've already tried."* The user explains for twenty seconds: tried integration by parts, got messy, suspects substitution. The voice: *"Good — what would you pick as `u`?"* The user thinks aloud, writes `u = x²` on the paper, holds it back up. The voice sees it and responds: *"That's the choice I'd make. What's `du`?"* The user works out `du = 2x dx` on the paper. The voice: *"You've got `2x dx` on the right and `x dx` in the integral — what does the integral become in terms of `u` and `du`, once you handle that factor of 2?"* The user works it out: `½ ∫ sin(u) du`. The voice: *"That's it. Now finish."* The user writes `-½ cos(u) + C` and then `-½ cos(x²) + C`, holds it up. The voice sees the final answer, holds for a second, and says quietly: *"That's right. You did the substitution; I just kept you company. Want me to save the transcript?"* The user walks away with the problem solved by *them*. The transcript saves to a quiet feed — the photograph of their work, the Socratic dialogue beside it, the time the session took, the hint count (zero this time), and the moment the model heard them arrive. **Tagline:** _Point your camera at what you're stuck on. Get a calm voice that asks the next question — never gives the answer._ ## 2. Target audience The job-to-be-done is universal: *"I am stuck on a thing and I want someone to walk me through it without robbing me of the satisfaction of solving it."* That job shows up everywhere. - High-school and university students stuck on a problem set — algebra, calculus, physics, chemistry stoichiometry, organic- chemistry mechanisms, statistics, micro-economics graph problems - Junior developers debugging code — a React hook in an infinite loop, a Python list-comprehension returning the wrong shape, a Rust borrow-checker error, a SQL join with NULLs in the wrong places, an `await` that should have been a `Promise.all` - Adults learning new manual skills — a sourdough recipe at the "windowpane test" step, the gluten-free conversion of a familiar cake, the first time deboning a chicken thigh, the first time blanching tomatoes for sauce, knife sharpening on a whetstone - Excel / Sheets users with a broken formula — a `#REF!`, an `#N/A` in a VLOOKUP that should not be there, an array formula returning a single value, a pivot table double-counting - DIY at home — assembling IKEA furniture with a confusing diagram, re-grouting a bathroom tile, hanging a picture without a stud- finder, replacing a flapper valve in a leaky toilet cistern, resetting a tripped RCD on a consumer unit - Small-engine and appliance repair — the lawnmower that won't start, the dishwasher with a clogged drain pump, the washing machine with the F08 error code, the bicycle derailleur that won't shift into top gear - Music students working out a chord voicing, a fingering, a bowing — guitar, piano, violin, ukulele, traditional instruments - Language-learners reading a real-world sign, menu, or document they want to understand without having it translated wholesale - Anyone facing an instruction sheet, a manual, a recipe, a tax form, a knitting pattern, a sewing pattern, an LED-strip wiring diagram, an electrical-panel label — they want to be *taught* through it, not given the answer - Parents helping kids with homework who want to model "ask questions, don't give answers" — the app demonstrates the posture they want to learn themselves ## 3. Core value propositions Surface these clearly through copy, visual emphasis, and section ordering — they are the reasons the user picks this over the "give me the answer" alternatives. - **The camera is the input.** You do not type a problem into a textarea. You do not paste an error. You hold the phone over the paper, the laptop screen, the carburettor, the bookcase, and the model SEES what you see. Live API on Gemini 3.1 Flash Live takes the video frames at one frame per second and the audio stream together, and reasons across them at conversational latency. - **Socratic by design.** The model never solves the problem in one shot. It asks what you have already tried. It prompts you to the next step. It waits while you work. It celebrates when you arrive. The system instruction enforces this; the UI reinforces it; the hint budget visible at the top of the session keeps it honest. - **Real-time vision + voice.** Post-I/O 2026 the Live API's acoustic nuance is the headline — pitch and pace that feel human, not robotic. Combined with the video-frame channel, the experience is "a calm tutor leaning over your shoulder", not "a chatbot reading a transcript". This is the capability that did not exist before May 2026. - **One hint, then another, never the answer.** A hint budget (default 3 per session) is visible at the top. The model will not exceed it without you explicitly tapping *"I'm really stuck — give me a bigger hint"*. Even then it gives you a bigger hint, not the answer. The Socratic discipline is a feature, not a side-effect. - **The transcript is the artefact.** At the end of every session you have a saved record: the photograph of your work, the Socratic dialogue beside it, the timestamps, the hint count, the moment you arrived. Reviewable later. Shareable with a classmate. Searchable across all your sessions ("when did I last work on integration by substitution?"). - **For adults learning, the rule is the same.** Whether it is a calculus problem or a carburettor float bowl, the model never paste-solves. For code the user types the next line themselves — the model describes what to type, not what to copy. For a recipe the user does the step themselves — the model describes the texture/colour/feel to look for. The dignity of solving it yourself is preserved across domains. - **Honest about uncertainty.** If the model cannot see the problem clearly through the camera (glare, focus, partial view), it says so calmly and asks you to adjust — it does not guess. If the problem is outside its competence (a graduate- level proof, a specialised industrial machine), it says so rather than fabricating a method. - **Hard rule on professional advice.** The tutor mode is for learning. Anything that smells like medical, legal, electrical- safety, gas-safety, or structural-safety advice surfaces a not-a-substitute banner. The model can ask the user what they see and ask them what they have considered — it does not diagnose, it does not authorise, it does not say "this is safe". The disclaimer is not an afterthought; it is part of the contract. ## 4. Features to build - **Start a session** — one big record button on the home screen; picks up the device camera and microphone, opens a Live API websocket session with `gemini-3.1-flash-live-preview`, streams audio in (PCM 16k) and video frames at 1 FPS, plays audio out (PCM 24k) through the device speaker or the user's headphones. - **Subject hint chips** at session start — *Maths*, *Code*, *Recipe*, *Spreadsheet*, *DIY / repair*, *Music*, *Language*, *Other*. Each chip nudges the session-system-prompt's domain vocabulary; none of them changes the Socratic posture. - **Hint budget visible at the top** — three filled dots by default; one dot empties each time the model offers a hint; when zero are left, the model defaults to encouragement rather than additional hints, and the user can tap *"I'm really stuck"* to request a bigger (but still not full-answer) hint. - **Live transcript pane** below the camera preview — caption streams the model's spoken words in real time; the user's spoken words appear in a different style (less prominent, the user does not need to read their own words). The transcript is scrollable mid-session if the user wants to reread. - **"Show me what you see"** toggle — when on, the model occasionally pauses to describe what it sees in the camera ("I see a handwritten integral, `∫ x · sin(x²) dx`, and you've written `u = x²` in the margin"). Useful for blind / low-vision users; also useful when the user wants the model to confirm framing before they start working. - **"I'm stuck" button** — taps add one hint to the budget for this session only (max +2 over the session). The model acknowledges the extra latitude calmly and gives a slightly bigger hint, never the answer. - **"I think I've got it" button** — the user taps when they believe they have the answer; the model asks them to read it out loud or hold up their working, and confirms or asks a follow-up question. This is the "moment of arrival" the transcript marks. - **Pause / Resume** — clean pauses on the Live API session (closes the audio channel but preserves session state for up to two minutes). Useful when the user needs to write quietly without the model listening. - **End session + save transcript** — closes the Live API session; the server writes the transcript to Firestore, the final photograph (last frame the model commented on) to Firebase Storage, the hint count, the duration, the moment-of- arrival timestamp. - **Session library** — a chronological feed of past sessions, each one a card with the subject, the photograph of the work, the first line of the transcript, the moment-of-arrival, and the hint count. Searchable. - **Session detail view** — full Socratic transcript beside the photograph(s) of the user's working. Each model line timestamped. Each user line timestamped. The hint-budget history is shown as a small timeline at the bottom ("3 hints remaining → 2 hints remaining at 1:42 → 2 hints remaining, moment of arrival at 4:18"). - **Resume an unfinished session** — if the Live API session hit the 2-minute audio+video cap before the user arrived, the session is saved as "in progress" and can be resumed with a new Live API connection plus a short text-only summary of where the previous segment left off (passed as system context). - **Subject-specific affordances:** - *Code mode*: the camera frames a code editor; the model reads what is on screen and asks the user to type the next line themselves — never pastes a solution. The transcript captures the code snippets verbatim from the OCR. - *Recipe mode*: the model describes textures and colours rather than minutes ("the dough should pass the windowpane test — stretch a small piece thin between your fingers; if you can see light through it without tearing, you're there"). - *DIY mode*: the model surfaces a "not-a-substitute" banner whenever the user shows electrical work above 12V, gas components, load-bearing structural work, or anything with safety implications. The model asks; it does not authorise. - *Maths mode*: the model resists arithmetic temptations — it never computes; it asks the user what they get. - **Reduced-motion + calm-acoustic mode** — toggle in settings; reduces the model's pace by ~15% and removes intonation flourishes, for users who find expressive TTS distracting. - **Family-mode safety wrapper** — when enabled, the session refuses to engage with any subject outside *Maths, Code, Recipe, Spreadsheet, DIY, Music, Language* and refuses any off-topic conversation, so a kid using the parent's device stays in the homework lane. - **Practice session library** — a small built-in library of starter problems per subject (one algebra problem, one React-hook bug, one sourdough step, one Excel formula error, one IKEA bookcase, one bicycle derailleur). Useful for the first-visit demo, useful for parents wanting to see the posture before handing the phone to a kid. - **Export transcript** — Markdown export for code sessions, PDF export for maths sessions (with the photograph of the working inline), email-to-self export for any session. - **Calendar reminders** — local reminders only, never push from the server. "Revisit the calculus session from Tuesday before the Friday quiz" is the right shape. ## 4b. Required Gemini capabilities + backend services **This template's intelligence comes from the Gemini capabilities below. Wire them up explicitly — don't substitute generic LLM calls.** ### Gemini capabilities (the load-bearing intelligence) - **Live API session** (`gemini-3.1-flash-live-preview`) — the hero capability. Streams audio in (PCM 16k mono) and video frames at ≤1 FPS; streams audio out (PCM 24k mono); takes text turns mixed into the same channel for things like the session preamble. Acoustic nuance (pitch, pace) was the I/O 2026 lift — sentences sound considered, not robotic. **Hard limits**: 15 min for audio-only sessions, **2 min for audio + video sessions**. Function calling is **synchronous only** — the model pauses speaking while a function returns. The session can be resumed with ephemeral session tokens. - **Live API function calling** — three synchronous functions are wired: - `markHint()` — the model declares it just offered a hint; the client decrements the hint budget and re-renders the dots. The model must call this any time it gives more than a Socratic question, so the user sees the budget shrink. - `markArrival(answerSummary, confidence)` — the model declares that the user has arrived at the answer; the client logs the moment-of-arrival timestamp, prompts the user to hold up their work, and offers the *Save transcript* action. - `requestStillFrame()` — the model asks the client to grab a high-resolution still (1080p) of the current camera view because the 1-FPS stream is too compressed to read fine detail (handwriting, code, recipe text, error codes). The client uploads the still to the Gemini Developer API Files API and the model references the resulting `files/*` resource name (e.g. `files/abc123xyz`) via `fileData.fileUri` in the next turn. - **Pre-session subject vocabulary primer** (Gemini 3.5 Flash, thinkingLevel `low`) — when the user picks a subject chip, a short text-only call assembles a `SessionPreamble` JSON that the client prepends as the first system message into the Live API session. The preamble names the subject's vocabulary, the domain-specific Socratic patterns, and any safety flags (e.g. "if user shows mains electrical, surface disclaimer"). - **Transcript post-processor** (Gemini 3.5 Flash, thinkingLevel `low`) — after a session ends, takes the raw turn-by-turn transcript + the captured still frames and emits a clean `SessionTranscript` JSON: the Socratic dialogue, the hint events, the moment-of-arrival, the topic summary, the next- session-suggestion (one short follow-up problem the user could try). Structured output via `responseSchema`. - **Subject classifier** (Gemini 3.5 Flash, thinkingLevel `low`) — if the user does not pick a subject chip and just starts a session, the first ~10 seconds of video frames go to a classifier call that returns one of `maths`, `code`, `recipe`, `spreadsheet`, `diy`, `music`, `language`, `other` plus a confidence score. The result feeds the preamble. If confidence < 0.6 the client shows the chips to ask explicitly. - **Image still understanding for the transcript** (Gemini 3.5 Flash, multimodal image) — the Live API stills captured via `requestStillFrame()` are also fed back into the post- processor so the transcript can include readable OCR of code, legible maths working, and recipe text. Used post-session, not in the live loop (the live loop has Live API's own vision). - **Subject-mode safety classifier** (Gemini 3.5 Flash, thinkingLevel `low`) — runs on every still frame the model captures via `requestStillFrame()`; returns a small JSON flagging whether the still shows mains-electrical, gas, load-bearing structural, medical/wound, firearm, or other professional-territory content. If flagged, the client surfaces the not-a-substitute banner and the next Live API turn injects a system message reminding the model not to authorise the work. - **Hero / empty-state illustration** (`gemini-3.1-flash-image`, Nano Banana 2) — generates the welcome and empty-state imagery: a phone in a stand on a kitchen table at golden hour with a notebook open showing a calculus problem, soft warm light, no people. - **Demo voice samples for the audio-preview** — recorded once via `gemini-3.1-flash-tts-preview` and cached as static MP3 in `/public/audio-samples/`; the landing page plays them so a visitor can hear the tutor voice before starting a real Live API session. **Do not use 2.5-flash-preview-tts** — that is the pre-I/O preview and has been superseded. - **Thinking levels** — `low` for every text-only call in this template (subject classifier, preamble, transcript post- processor, safety classifier). The Live API model (`gemini-3.1-flash-live-preview`) does not take a `thinkingConfig` field — omit it entirely. The TTS and image- generation calls also do not take `thinkingConfig`. ### Backend services - **Auth — Required.** Firebase Auth with Google sign-in (auto- provisioned by AI Studio Build). Apple sign-in is optional and user-configured (requires Apple Developer account, Service ID, Key ID, private key in the Firebase Auth console). Magic-link email (used to email a saved transcript to the user) requires the sender domain to be authorised in Firebase Auth. - **Database — Required.** Firestore for `users`, `sessions`, `transcripts`, `still_frames_metadata`, `practice_problems`, `family_mode_settings`. Sessions are private to the owner by default. - **File storage — Required.** Firebase Storage for the captured still frames (the `requestStillFrame()` images) and for any recorded audio the user chooses to save. **Storage is NOT auto-provisioned by AI Studio Build today** — enable it in the Firebase console and wire the bucket name into the AIS Build project before the first session runs. Pre-signed URLs only. - **Live API gateway — Required.** A Cloud Run server function (auto-provisioned) opens the Live API websocket connection using the project's `GEMINI_API_KEY` from Secrets Manager and proxies frames + audio between the client and Google. **Do not expose `GEMINI_API_KEY` to the browser.** Use ephemeral session tokens issued by the gateway so the client only ever holds a short-lived token, not the API key itself. The Cloud Run free 2-app deploy (post-I/O) covers this app's gateway out of the box. - **Email — Required (transactional).** Magic-link share email for transcripts the user chooses to email to themselves or to a classmate. Sender domain authorised in Firebase Auth. - **Payments — Not needed for v1.** Free for personal use. Future "Family Pro" tier (multi-child sub-accounts, parent digest emails, longer hint budgets per subject) might charge via Stripe; not built in v1. - **Workspace integration — Optional.** Post-I/O 2026 Workspace integration without OAuth handshake — for users who want transcripts auto-saved to Google Docs / Drive, this is a one- toggle setting and no longer requires a per-user OAuth consent flow. Verify availability in your AIS Build configuration before promising it. - **External APIs:** Gemini API for all intelligence (Live API, Flash text calls, image generation, TTS for cached samples). No required external API beyond Gemini. **Environment variables:** every secret (Gemini API key, Firebase service-account JSON, Stripe key if pro tier added) lives in environment variables — never in client bundle. Include a `.env.example`. **Auth + data privacy reminders:** never log secrets · never store passwords in plain text · use HTTPS everywhere · honour 'delete my account' inside the UI · explicit opt-in for any analytics · the user's session audio, video frames, and transcripts are never sent to Gemini for model training (use the Gemini API on the paid tier, where Google does not use your content for model training, per the Gemini API Additional Terms) · family-mode sessions are scoped to the family unit and never indexed. **Read this first — prompt-craft rules that apply to every call in this template:** 1. **Name the model variant explicitly** in every Gemini API call. Do not let the agent pick the model. See the per-call matrix below. 2. **Pin `thinkingLevel` explicitly** per call. See the matrix. Models that don't take a thinkingConfig (the Live API model, the image-generation model, the TTS model) — omit the field entirely. 3. **Seed the JSON Schema as a fenced TypeScript / Zod block** in the system instruction or `responseSchema` field. The literal schemas are below. **Convert the Zod schema to Gemini's `Schema` type via the SDK helper** before passing to `responseSchema` — do NOT pass raw Zod. **Numeric `min`/`max` constraints are documentation only inside `responseSchema`; clamp on the server after the response arrives.** 4. **Pin the system instruction separately** from user input. Use the `systemInstruction` field for persona + behavioural rules; use `contents` for user input. Never concatenate. 5. **Pre-declare tools as an enable/disable list** per call. The matrix below names which tools are enabled per call. Tools NOT listed for a call should be disabled. 6. **State negative constraints explicitly** — they are listed below. They are NOT "be careful" suggestions; they are hard rules the model must follow. 7. **Live API function calling is synchronous only.** The model pauses speaking while a function call returns. Keep function handlers fast (<200ms) — long ops should be deferred to a post-turn worker, not blocked on inside `markHint` or `markArrival`. 8. **Live API video+audio sessions cap at 2 minutes.** Build resumption explicitly: at 1:45 the client warns the user, at 1:55 the client gracefully closes and queues a resume token. 9. **Files API uses `files/*` resource names, not `gs://` URIs.** The AI Studio Build runtime uses the Gemini Developer API (`@google/genai` SDK). Files API `upload` returns a resource name of the form `files/abc123xyz`, passed via `fileData: { fileUri, mimeType }`. `gs://` URIs belong to Vertex AI / Cloud Storage — a different surface, not accepted here. 10. **Grounded responses can wrap JSON in ```json fences or add prose preamble** (the next-session-suggestion call uses `google_search`). Server-side, strip fences and brace-extract: ```typescript function safeExtractJSON(raw: string): T { const clean = raw.replace(/```json\s*|```/gi, '').trim(); const s = clean.indexOf('{'); const e = clean.lastIndexOf('}'); if (s === -1 || e === -1) throw new Error('No JSON boundaries in grounded response'); return JSON.parse(clean.slice(s, e + 1)) as T; } ``` 11. **Strip unsupported Zod modifiers before passing to `responseSchema`** — Gemini's OpenAPI subset rejects `.regex()` / `pattern`, fixed-length `z.tuple()`, and other custom validators. Use a sanitizer that flattens tuples to arrays and removes regex patterns before serializing. Validate those constraints in middleware AFTER parsing. ### Per-call model + tools matrix | Call | Model | thinkingLevel | Tools enabled | |------|-------|---------------|---------------| | Live session (audio in + video frames in + audio out) | `gemini-3.1-flash-live-preview` | n/a | function calling: `markHint`, `markArrival`, `requestStillFrame` | | Subject classifier (first ~10s of frames) | `gemini-3.5-flash` | low | (none) — pure multimodal image input | | Session preamble assembly (subject → SessionPreamble JSON) | `gemini-3.5-flash` | low | (none) | | Still-frame safety classifier (per captured still) | `gemini-3.5-flash` | low | (none) | | Transcript post-processor (raw turns + stills → SessionTranscript) | `gemini-3.5-flash` | low | (none) | | Next-session suggestion (one follow-up problem) | `gemini-3.5-flash` | low | `google_search` grounding (no `responseSchema` on this call — see note) | | Hero / empty-state illustration | `gemini-3.1-flash-image` (Nano Banana 2) | n/a | n/a | | Cached voice samples for the landing page | `gemini-3.1-flash-tts-preview` | n/a | n/a | *Note for builders:* the Live API call does NOT take a `thinkingConfig`; omit it. The image-gen and TTS calls also do NOT take a `thinkingConfig`. The grounded next-session- suggestion call cannot combine `responseSchema` with `google_search` in the same Gemini call today — instruct the model to emit JSON in the text body and parse it server-side; read citation URLs from `response.groundingMetadata.groundingChunks[].web.uri`. **Do not use `gemini-3.5-flash`, `gemini-3.5-flash`, or `gemini-2.5-flash- preview-tts`** — those are pre-I/O 2026 strings and now resolve to deprecated/missing endpoints. ### Primary structured-output schemas (seed verbatim in the prompt) ```typescript import { z } from "zod"; const SubjectMode = z.enum([ "maths", "code", "recipe", "spreadsheet", "diy", "music", "language", "other", ]); const HintEvent = z.object({ hint_index: z.number().min(1).max(5), // 1..3 base + up to 2 extra given_at_iso: z.string(), hint_size: z.enum(["small", "medium", "bigger"]), hint_text_summary: z.string(), // one sentence, not the verbatim hint user_was_stuck_for_seconds: z.number().nullable(), }); const SafetyFlag = z.object({ category: z.enum([ "mains_electrical", "gas", "load_bearing", "medical_wound", "firearm", "chemical_handling", "child_safety", "other_professional", ]), detected_in_still_uri: z.string(), surfaced_disclaimer: z.boolean(), }); const SessionPreamble = z.object({ subject_mode: SubjectMode, vocabulary_anchors: z.array(z.string()), // domain words the model will use socratic_patterns: z.array(z.string()), // questions to ask, never to answer safety_watchlist: z.array(z.string()), // categories to flag if seen initial_greeting_one_line: z.string(), // what the model opens with }); const TranscriptTurn = z.object({ speaker: z.enum(["model", "user"]), spoken_at_iso: z.string(), text_verbatim: z.string(), was_hint: z.boolean(), // true if this model turn was a hint references_still_uri: z.string().nullable(),// if the model referenced a captured still }); const SessionTranscript = z.object({ session_id: z.string(), subject_mode: SubjectMode, started_at_iso: z.string(), ended_at_iso: z.string(), duration_seconds: z.number(), hint_budget_at_start: z.number().min(1).max(5), hints_used: z.array(HintEvent), hint_budget_remaining: z.number().min(0).max(5), arrival_event: z.object({ arrived_at_iso: z.string().nullable(), // null if user did not arrive in this session arrival_summary: z.string(), // one sentence describing the answer arrival_confidence: z.number().min(0).max(1), }), safety_flags: z.array(SafetyFlag), still_uris: z.array(z.string()), // Files API `files/*` resource names of stills captured turns: z.array(TranscriptTurn), topic_summary_one_line: z.string(), // for the session library card next_session_suggestion: z.object({ suggestion_text: z.string(), // a follow-up problem the user could try suggestion_source_authority: z.string().nullable(), // if grounded, the source }).nullable(), }); const SubjectClassification = z.object({ inferred_subject_mode: SubjectMode, confidence: z.number().min(0).max(1), visible_artefacts: z.array(z.string()), // what the classifier saw needs_explicit_chip: z.boolean(), // true if confidence < 0.6 }); type SessionPreamble = z.infer; type SessionTranscript = z.infer; type SubjectClassification = z.infer; ``` ### Common failure modes (and how to avoid them) - **The model gives the answer.** The single most common failure. The Live API system instruction MUST forbid giving the answer outright; the `markHint()` function call records every hint so the client can enforce the budget; the post- session transcript post-processor counts hint events and flags any session where the model crossed the budget. If the budget was crossed, log to telemetry so the prompt can be tightened. - **The model goes longer than 2 minutes.** Audio+video sessions cap at 2 min on Live API. The client must warn at 1:45, save state at 1:55, and offer **Resume** which opens a new Live API connection with the previous session's transcript summary as a fresh system message. Do NOT promise "continuous" sessions. - **The Live API model picks the wrong subject vocabulary.** Without the `SessionPreamble`, the model defaults to general- purpose tutoring and uses awkward phrasing for code or recipes. Always run the subject classifier (or use the user's chip pick) and inject the preamble as the first system message into the Live API session. - **A safety-flagged frame slips through silently.** The still-frame safety classifier runs on every captured still and the client must surface the not-a-substitute banner before the next Live API turn. The next turn's system message must include "user is showing mains-electrical work; do not authorise; ask what they see and ask what a qualified electrician would check". - **The model recomputes maths/arithmetic.** The Live API model is a fast multimodal language model, not a calculator. It should never compute; it should always ask the user what *they* get. Negative constraint enforced in the system instruction. - **The model paste-solves code.** Forbidden. In code mode the model asks the user to type the next line; if the user asks "what's the line?" the model describes what the line does ("a `useEffect` that depends on the value you want to read but not write — what would you put in the dependency array?"), not the literal characters. - **The transcript post-processor invents turns that did not happen.** Hard rule: the post-processor receives the raw turn-by-turn audio transcript from the Live API and must only re-format / clean / summarise — never fabricate. If a turn is unclear in the audio, mark `text_verbatim` with `[unclear]` rather than guessing. - **Subject classifier mislabels a recipe as DIY.** A baking recipe shown on a tablet next to a kitchen counter can read like DIY at low frame rates. If classifier confidence is below 0.6 the client MUST show the chips for the user to pick explicitly, never silently default to the wrong mode. - **Live API ephemeral token leaks via the browser console.** Use the Cloud Run gateway to mint short-lived tokens (e.g. 10-min TTL), bind them to the user's Firebase Auth UID, and rotate per session. Never log them to the browser console; never store them in `localStorage`. - **Family-mode child wanders off-topic.** When family-mode is on, the Live API system instruction enforces a strict subject scope. The model must politely redirect any off- topic input ("let's get back to the maths problem on your paper"). If the off-topic persists beyond two redirects, the session ends with a calm note and the budget is preserved. - **The user is blind / low-vision and the "show me what you see" affordance is off.** Default behaviour for accessibility users (detected via OS-level prefers-reduced-motion + screen- reader heuristics) is to enable "show me what you see" with a 30-second cadence so the model speaks visual descriptions proactively, not just on request. ### Negative constraints (hard rules) - Do NOT give the answer outright. Never. Not even when asked directly. The model's response to "just tell me the answer" is to offer one more hint, then ask the user to take the next step. If the user is genuinely stuck and the hint budget is exhausted, the model gracefully ends the session with a warm note rather than capitulating. - Do NOT compute arithmetic. The model asks the user what they get. Even simple arithmetic ("what's 2 × 3?") is the user's to do. - Do NOT paste solutions in code mode. The user types every line themselves. The model describes the next line in natural language ("the dependency array should be empty because we only want this to run once on mount") and waits. - Do NOT translate problems wholesale in language-learning mode. The model asks the user what they think a word means, prompts with a related word they might know, and confirms when they arrive — it does not just translate. - Do NOT authorise safety-flagged work. For mains-electrical, gas, load-bearing structural, medical wound, firearm, or chemical-handling work, the model asks the user what they see and what a qualified professional would check. The not-a- substitute banner is non-dismissable in the UI for the remainder of the session once a safety flag fires. - Do NOT moralise about the user's choice of work. If a user chooses to fix their own lawnmower, the app helps them Socratically; it does not lecture them on calling a professional. - Do NOT gamify learning. No streaks. No badges. No "you've earned 500 XP". The arrival itself is the reward; the transcript is the artefact. - Do NOT log the user's full audio or video frames beyond the Live API session window. The raw video frames are streamed through the Cloud Run gateway and not persisted unless the user explicitly chose to save audio. Still frames captured via `requestStillFrame()` ARE persisted (they are part of the transcript artefact). - Do NOT use the user's session audio, video, or transcripts to train or fine-tune any model. Use the Gemini API on the paid tier, where Google does not use your content for model training, per the Gemini API Additional Terms. The capabilities-info panel says this in plain English. - Do NOT use a robotic monotone TTS. The acoustic nuance of Live API on 3.1 Flash Live is the headline experience — pace and pitch should match the moment (slower and softer when the user is stuck; brighter and faster on arrival). - Do NOT exceed the hint budget without the explicit "I'm really stuck" tap. The function call discipline enforces this — every hint emits `markHint()` and decrements the budget client-side. - Do NOT ship without the food-safety / professional-territory not-a-substitute banner wired up. The safety classifier runs on every captured still; the disclaimer is required, not optional. ### Per-call `systemInstruction` strings Use these as the literal `systemInstruction` field for each Gemini API call the built app makes. ### Call: Live API session (the core tutor loop) Model: `gemini-3.1-flash-live-preview` · thinkingLevel: n/a · Tools: function calling (`markHint`, `markArrival`, `requestStillFrame`) ``` You are a calm, patient tutor. You sound like a kind older sibling or a thoughtful coach. You are NEVER a chatbot, never a search engine, never a stack-overflow paste. You are a voice that asks the next question. You can SEE what the user's camera sees (video frames at ~1 frame per second) and you can HEAR what the user says (streaming audio). You speak back to the user in audio. YOUR JOB: 1. When the session starts, greet the user briefly (one short sentence) and ask what they're working on. If you can already see what's on the camera (a maths problem on paper, a code editor with an error, a recipe step, a piece of furniture being assembled, a small engine with the housing open, a musical instrument with the user's hands on it, a sign in a foreign language), name it specifically in your greeting ("I can see the integral ∫ x · sin(x²) dx — walk me through what you've already tried"). 2. ASK what the user has already tried before suggesting anything. Do not jump to a hint until you've heard their approach. If the user says "nothing yet", that's fine — suggest they start by looking at the problem and naming what kind of thing it is (a substitution problem, a syntax error, a missing ingredient step, a loose cam-lock). 3. Move the user FORWARD by one step at a time. Each of your turns is a question or a small Socratic prompt, never a solution. "What would you pick as u?" is good. "Pick u = x²" is too much. "The substitution is u = x²; then du = 2x dx; then the integral becomes ½∫sin(u)du" is forbidden. 4. Use the user's own vocabulary. If they say "the squiggly thing" point to it in the camera and name it ("the integral sign"); if they say "the parens", say "the parens"; if they say "that bit", ask "the part where you wrote the number?" — match their language. 5. CELEBRATE quietly when the user arrives. A single calm line ("That's it. You did the substitution; I just kept you company"). No fireworks, no exclamation salad. WHEN YOU GIVE A HINT (and you should give few): - Call the `markHint()` function so the client can decrement the visible hint budget. Pass the hint size: "small" (a pointing-finger question), "medium" (naming the technique but not the steps), "bigger" (walking through the first step's setup without giving the result). - A "small" hint is the default. Move from small → medium → bigger only across turns, never within one turn. - After three hints (the default budget), default to encouragement and reflection ("you've gotten this far — what's left to check?"). Do not offer a fourth hint unsolicited. The client will tell you if the user has tapped "I'm really stuck" — then you can offer one bigger hint, but never the answer. WHEN YOU CAN'T READ THE CAMERA CLEARLY: - Call `requestStillFrame()` to grab a 1080p still you can read. Wait for the client to confirm; then in your next turn reference what you can now see. - If the still is still unclear, say so calmly and ask the user to reposition, improve lighting, or move closer. Do NOT guess. WHEN THE USER ARRIVES AT THE ANSWER: - Confirm their answer with one Socratic check: "read it back to me" or "hold up the page so I can see". Do not declare arrival from a glance alone. - When confirmed, call `markArrival(answerSummary, confidence)` so the client can save the moment-of-arrival timestamp. `answerSummary` is one sentence in the user's own framing; `confidence` is your honest read of whether they really got there. WHEN THE WORK SHOWS SAFETY-PROFESSIONAL TERRITORY (mains electrical, gas, load-bearing structural, medical wound, firearm, chemical handling): - The client will inject a system note when a safety flag fires. From that point in the session you do NOT authorise the work. You can still help the user think Socratically about what they SEE and what a qualified professional would check. Do not say "this is safe" or "this is fine to do yourself". Do say "what would an electrician / plumber / structural engineer look for here?". - If the user asks you directly "is this safe", your answer is "I can help you think through what to check, but I'm not qualified to say it's safe — that's what an electrician / plumber / clinician / structural engineer is for". HARD RULES: - Never give the answer. Never. If asked directly, offer one more hint and ask for the next step. - Never compute arithmetic. Ask the user what they get. - Never paste code. In code mode, describe what the next line does; the user types it. - Never translate wholesale in language mode. Prompt with a related word the user might know. - Never moralise. Never lecture. Never gamify. - Never use exclamation salad ("Great job!! Awesome!! You did it!!"). One quiet, sincere line at arrival. - Never go off-topic in family-mode (the client will tell you when family-mode is on). If the user wanders, redirect calmly once. If they wander again, redirect again. If they wander a third time, end the session warmly. VOICE: - Pace: unhurried; pauses where a thoughtful person would pause; slower and softer when the user is visibly stuck; brighter and a touch faster on arrival. - Acoustic nuance: this is the post-I/O 2026 Live API upgrade. Use it. Sound considered, not robotic. - Length: most turns are one sentence. Two when a Socratic prompt needs framing. Three only when celebrating arrival. If at any point you do not know how to help (a graduate- level proof outside your competence, a specialised industrial machine, a language you cannot parse), say so honestly: "this is outside what I can help you with — let me suggest where you might go next". Do not fabricate. ``` --- ### Call: Subject classifier (first ~10 seconds of video frames) Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none) ``` You receive 5-10 video frames captured at 1 FPS from the user's camera at the start of a Live Tutor session. The user has not yet picked a subject chip. Your job is to classify which subject mode best fits what is visible. Subject modes: - maths — handwritten or printed mathematics problems on paper or a tablet (algebra, calculus, statistics, etc.) - code — a code editor, terminal, or IDE visible on a laptop or monitor screen - recipe — a kitchen or cookbook scene; ingredients, mixing bowls, dough, a printed or screen-displayed recipe - spreadsheet — a spreadsheet visible on a laptop or monitor (Excel, Sheets, Numbers); formulas, error codes (#REF!, #N/A, #DIV/0!) visible - diy — household repair or assembly; furniture being assembled, tile work, plumbing, electrical, small-engine parts, bicycle components - music — a musical instrument with the user's hands on it (guitar, piano, violin, etc.); sheet music; tabs on a screen - language — a foreign-language sign, menu, document, or textbook - other — none of the above; or genuinely ambiguous Output the SubjectClassification JSON exactly. Set needs_explicit_chip: true whenever confidence < 0.6. Hard rules: - Do NOT classify based on the user's clothing, room style, or other irrelevant cues. Classify ONLY based on the work artefact visible. - visible_artefacts is a short list of concrete things you saw ("handwritten integral with x squared", "VS Code with a useEffect hook", "stainless mixing bowl with dough"). - If multiple subjects are present (e.g. a laptop with code next to a maths textbook), pick the one most likely to be the active subject and lower confidence accordingly. - Confidence of 0.9+ requires unambiguous evidence in at least 3 of the frames. Output: SubjectClassification JSON. No commentary. ``` --- ### Call: Session preamble assembly Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none) ``` You receive a subject mode (one of: maths, code, recipe, spreadsheet, diy, music, language, other) and optionally a short user-supplied note (e.g. "I'm a Year 12 student working on differentiation"). Output a SessionPreamble JSON that will be injected as the first system message into the Live API session. Fields: - subject_mode: copy the input. - vocabulary_anchors: 5-12 domain-specific words / phrases the model should use in this session ("integral", "substitution", "u-substitution", "antiderivative" for maths; "useEffect", "dependency array", "stale closure", "memoization" for code; "windowpane test", "autolyse", "bulk ferment", "shaping", "proofing" for recipe / baking). - socratic_patterns: 4-6 Socratic question patterns suited to this domain ("What would you pick as your substitution variable?", "What does the dependency array control in a useEffect?", "What does the dough feel like right now?"). Patterns are templates, not full questions. - safety_watchlist: any safety categories worth flagging if seen in subsequent frames (e.g. for DIY mode: mains electrical, gas, structural; for recipe mode: raw meat handling, hot oil, knife use; for code mode: none unless the code shows credentials). - initial_greeting_one_line: a single short greeting the Live API model can use to open the session (e.g. "I can see your integral — walk me through what you've already tried"). Keep it specific to the subject and warm. Hard rules: - Vocabulary anchors must be REAL terms in the domain, not invented. If you don't know the term, omit it. - The greeting must be specific to the subject and must NOT give any hint about the problem itself. "I can see your integral" is fine; "I see u-substitution will work here" is forbidden. - Do NOT translate vocabulary anchors. "u-substitution" stays in English even if the user's interface is in French. Output: SessionPreamble JSON. No commentary. ``` --- ### Call: Still-frame safety classifier Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none) ``` You receive a single still frame captured during a Live Tutor session via the model's requestStillFrame() function call. Your job is to flag whether the frame shows any professional-territory subject matter where the app must surface a not-a-substitute disclaimer and where the Live API model must not authorise the work. Categories to flag: - mains_electrical — anything 50V or above, consumer-unit panels, RCDs, wiring inside a wall plate, ring-main components, electric showers, EV chargers - gas — gas hobs / boilers / cylinders / regulators / flues - load_bearing — structural walls being modified, lintels, beams, joists being cut - medical_wound — visible open wound, suture, blood, a body part the user appears to be treating - firearm — any firearm or major firearm component - chemical_handling — caustic / acidic / hazardous chemical containers (drain cleaner, pool acid, paint stripper) - child_safety — a young child appears to be operating the tool or handling the dangerous object themselves - other_professional — anything that looks like professional-only territory (industrial machinery, scientific apparatus, certified equipment) Output JSON: { "flags": [SafetyFlag, ...], // per the schema; can be [] "neutral_description_one_line": "" } Hard rules: - Do NOT diagnose. "There is a wound visible on the user's hand" is a description; "the user has a laceration that needs sutures" is a diagnosis. Never diagnose. - Do NOT guess. If you cannot tell whether the wire in frame is mains or low-voltage, do not flag — but do describe what you see neutrally. - Multiple flags per frame are fine. - surfaced_disclaimer: true means the client should show the banner. Default true for any flag. Output: JSON. No commentary. ``` --- ### Call: Transcript post-processor Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none) ``` You receive the raw turn-by-turn transcript from a Live API session (audio transcribed by Gemini live, with speaker labels "model" and "user"), plus the list of Files API `files/*` resource names of stills captured during the session, plus the hint events recorded by the markHint() function calls, plus the markArrival() event if present. ⚠️ Strict Token Preservation: when transcribing or re-rendering handwritten mathematics from a captured still, map directly to canonical LaTeX (e.g. `\\int x \\cdot \\sin(x^2) \\, dx`). Do NOT prose-describe the expression and do NOT silently normalise notation. If any expression is ambiguous, set `confidence: 'low'` + `flagged_for_user_review: true` for that turn instead of guessing. Your job is to emit a SessionTranscript JSON that is the saved artefact of the session. What you do: - Clean each turn's text_verbatim: remove "um" / "uh" / audible breath markers, but PRESERVE the user's actual words and the model's actual phrasing. Do NOT paraphrase. - For each model turn, set was_hint: true if a markHint() event was emitted during that turn. - For each model turn that referenced a still, set references_still_uri to the relevant Files API `files/*` resource name. - Compose topic_summary_one_line: one sentence the user can read in the session library card ("Worked through integration by substitution on ∫ x · sin(x²) dx — 1 hint used, arrived in 4:18"). - Set hints_used to the array of HintEvents (with hint_index, given_at_iso, hint_size, hint_text_summary, user_was_stuck_for_seconds). - Set arrival_event from the markArrival() data if present; otherwise set arrived_at_iso to null and arrival_summary to "user did not arrive in this session". Hard rules: - Do NOT fabricate turns. The transcript reflects only what was actually said. If a turn was unclear in the audio, mark text_verbatim with "[unclear]" and an approximation in parens. - Do NOT score the user. The transcript is a record, not a report card. - Do NOT auto-add hints that were not flagged via markHint(). The model's discipline is the source of truth. - next_session_suggestion is set ONLY when arrival happened AND the user opted in (the client will tell you). Otherwise null. Output: SessionTranscript JSON. No commentary. ``` --- ### Call: Next-session suggestion Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: `google_search` grounding ``` You receive a SessionTranscript object for a session that ended with arrival. Your job is to suggest ONE follow-up problem the user could try in their next session, of slightly higher difficulty (or covering a nearby concept). Use google_search to ground the suggestion in a real textbook problem, a real well- known coding challenge, a real recipe variation, a real DIY follow-up — never invent a citation. Use `google_search` grounding for every suggestion. Output the suggestion as JSON in the text body (NOT via `responseSchema` — responseSchema and google_search cannot be combined in the same Gemini call today). The server will parse the JSON and read citation URLs from response.groundingMetadata.groundingChunks[].web.uri. JSON shape: { "suggestion_text": "", "suggestion_source_authority": "", "difficulty_note": "" } Hard rules: - The suggestion must be at most one small step up in difficulty. Do not pick a wildly harder problem just because the user arrived. - The suggestion must be specific. "Try a harder integral" is not specific; "Try ∫ x² · cos(x³) dx — same pattern, different exponent" is specific. - Do NOT include URLs in the JSON body. Citations come from groundingMetadata server-side. - If the user wandered into a domain outside the standard subject modes ("other" mode), suggestion may be null and that's fine. No commentary outside the JSON. ``` ## 5. Use cases & content to include Build dedicated UI flows or seed sessions for each of these — they tell you what content the app must support. - **The calculus problem at the kitchen table.** A student has `∫ x · sin(x²) dx` on a worksheet. They prop the phone in a stand, point the camera at the paper, tap **Start**. The model greets, asks what they've tried. They explain. The model asks "what would you pick as `u`?". They pick `u = x²`. The model asks what `du` is. They work it out. The model asks how the integral transforms. They get there in four minutes with one small hint. Transcript saves with the photographed working. - **The React `useEffect` infinite loop at 11pm.** A junior developer has a `useEffect` firing every render. They point the phone at the laptop screen showing VS Code. The model reads the code, asks "what's in your dependency array?" The developer realises they put the state setter in there. The model asks "what should be in there instead?". The developer types the fix themselves. The model never pastes a solution. - **The windowpane test.** A baker is trying their first sourdough. The recipe says "knead until the windowpane test passes" and they have never heard of it. They prop the phone on a stand over the dough, tap **Start**. The model asks what the dough feels like, asks them to tear off a small piece and stretch it. They do. The model asks what they see when they hold it up to the light. They see translucency without tears. The model says "that's the windowpane — you're there". - **The `#REF!` spreadsheet error.** A user has a VLOOKUP returning `#REF!` in cell `D14`. They show the spreadsheet to the camera. The model asks what the third argument of the VLOOKUP is (the column index). They say "5". The model asks how many columns the lookup table has. They count: "4". The user fixes it themselves and the model marks arrival. - **The wobbly Billy bookcase.** A new-flat dweller has a Billy bookcase assembled but wobbly. They show the camera the back. The model asks if the cam-locks at the bottom are tight. They check and find two loose. The model asks what tool they'd use. They get a flathead screwdriver. The model waits while they tighten. The bookcase stops wobbling. Arrival. - **The lawnmower that won't start.** A user has a petrol lawnmower that pulls but won't fire. They show the spark plug area, the air filter housing, the fuel tank. The model surfaces a not-a-substitute banner the moment it sees a small-engine carburettor (chemical_handling for fuel, professional territory for repairs) and continues Socratically: "when did you last change the spark plug?" The user replaces it themselves and the mower fires. Arrival, with the banner visible the whole time. - **The bicycle derailleur that won't shift.** A weekend cyclist has a rear derailleur skipping gears. They show the cassette and the derailleur. The model asks them to shift through the gears slowly while watching. They see the chain hesitating between two specific cogs. The model asks what the limit screws do. The cyclist remembers reading about high and low limits, finds the H screw, adjusts a quarter-turn. The chain settles. Arrival. - **The chord voicing on guitar.** A student is learning the F-major barre chord and can't get a clean sound. They show their hands on the fretboard. The model asks which strings are buzzing. They identify the B and high E. The model asks where their index finger is pressing on those strings. They look. The model never plays the chord for them; it asks the next question. They reposition their finger and get the clean sound. Arrival. - **The Spanish menu on holiday.** A traveller in Seville has a menu they want to understand without just translating everything. They show the camera "merluza a la plancha con piquillos". The model asks what they recognise. They get "a la plancha — grilled". The model asks what they think "merluza" might be (a fish). They guess hake. The model confirms with "you've got it — a white fish related to cod". They order with confidence. - **The family-mode kid working on long division.** A parent has switched family-mode on; their 10-year-old is working on `742 ÷ 7`. The kid asks the model "what's the answer?" The model calmly redirects: "let's work it out together. What's the first thing you do in long division?" The kid works through it. Arrival. ## 6. Page structure Build the following screens / sections in this order. Adjust copy to fit the voice, but keep the structural intent. 1. **Welcome / sign-in.** A photographed-looking shot of a phone in a stand on a kitchen table at golden hour, a notebook open showing a handwritten calculus problem, the phone showing the Live Tutor record screen. One paragraph: "Live Tutor is a calm voice that walks you through whatever you're stuck on. Point your camera. Get the next question — never the answer." Single Google sign-in button; Apple sign-in next to it. Below: a **Hear the voice** button that plays a 12-second pre-recorded audio sample of the tutor saying a sample Socratic exchange. Below that: **Try a practice session** → loads one of the demo problems from section 8a. 2. **Subject chips screen** (shown when the user taps **Start a session** from the home screen). Seven chips: *Maths*, *Code*, *Recipe*, *Spreadsheet*, *DIY / repair*, *Music*, *Language*. An eighth small text "Or just point your camera — I'll figure it out". Above the chips, the live camera preview at half- height so the user sees the framing before they tap. 3. **Active session screen.** Full-screen camera preview. At the top, the hint-budget dots (three filled), the session timer, a *Pause* button, and a small *Show me what you see* toggle. At the bottom, a live transcript pane (~1/3 height, scrollable mid-session). Floating above: *I'm stuck*, *I think I've got it*, *End session*. The not-a-substitute banner, when surfaced by a safety flag, appears as a slim pinned banner above the transcript pane; non-dismissable for the remainder of the session. 4. **Arrival moment.** When the model calls `markArrival()`, the screen briefly dims around a centred card: "You did it — want to save the transcript?". The card shows the most- recent still the model commented on, the topic-summary one- liner, the time taken, the hint count. Two buttons: *Save transcript* (primary), *End without saving* (secondary). 5. **Session library.** A chronological feed of past sessions. Each card: subject icon, topic summary, duration, hint count, the moment-of-arrival timestamp. Search bar at the top. Filter by subject chip. 6. **Session detail view.** Full Socratic transcript on the left half; the still frames captured during the session as a scrollable strip on the right. Each model turn timestamped; each hint event marked with a small dot in the margin. The moment-of-arrival highlighted. Buttons at the top: *Export PDF*, *Export Markdown*, *Email to myself*, *Resume this session* (only if the previous session ended at the 2-min cap without arrival). 7. **Practice session library.** A small built-in library of starter problems per subject. Each problem is a card with the problem statement, the subject chip, and a *Try this session* button that pre-loads a demo camera view (using a cached sample image) and a curated `SessionPreamble`. 8. **Family-mode panel.** Settings → Family-mode. Toggle. Strict-subject-scope toggle (defaults on). Hint budget per subject. Redirection style: gentle / firmer. Parent digest email frequency: never / weekly / never (default never). 9. **Settings & privacy.** Voice mode (default / calm-acoustic). Camera preview position (top / bottom). Default subject mode (off). Workspace integration toggle (post-I/O; off by default). "Delete my account" / "Delete a session forever" with a 60-second cool-off. Privacy panel restates the not- trained-on policy in plain English. 10. **Footer.** "Made for the moment you're stuck." Privacy: "Your sessions are yours. We never train on them." Capabilities `(i)` icon in header. ## 6b. First-visit onboarding Show a **first-visit onboarding** the first time a visitor lands on the app (detect via `localStorage` flag; do not show on return visits). Three slides, dismissible at any time. Persistent re-entry: a `?` icon in the header reopens it. **Slide 1 — What this is.** - Headline: "Welcome to Live Tutor." - Subhead: "Point your camera at what you're stuck on. Get a calm voice that asks the next question — never gives the answer." - One paragraph (≤ 60 words): "Live Tutor uses Gemini 3.1 Flash Live to see what your camera sees and hear what you say, in real time. It's Socratic by design — it asks what you've tried, prompts the next step, waits while you work, and celebrates when you arrive. Three hints per session, visible at the top." - Visual: an annotated illustration of the active session screen — hint-budget dots labelled, camera preview labelled, transcript pane labelled. Not a generic phone mockup. **Slide 2 — Try it now.** - Headline: "Hear the voice, try a practice problem." - A **Hear the voice** button — plays a 12-second pre- recorded sample of the tutor voice doing a Socratic exchange. (Cached MP3 generated via `gemini-3.1-flash-tts-preview`; not a live session.) - A **Try a practice problem** button — opens a session pre-loaded with one of the demo problems from section 8a (a calculus problem, a React-hook bug, a sourdough step, a Billy bookcase, or a Spanish menu). **Slide 3 — How to remix this.** - Headline: "Make this yours." - Three short bullets: - "Swap the practice problem library in `/data/practice/` for your own seed problems." - "Adjust the per-subject preambles in `/server/preambles/` to fit your tutoring style and vocabulary." - "Wire up your Gemini API key and Firebase project via the env-var list in the capabilities panel." - Primary CTA: "Use this template" → links to AI Studio Build remix entry point. - Secondary: "Just exploring — close" (sets localStorage flag, never auto-shows again). **Accessibility:** focus trap, `Esc` closes, `role="dialog"`, `aria-modal="true"`, `aria-labelledby`, focus restored to trigger on close. Respect `prefers-reduced-motion`. **Don't:** - Don't gate content behind the modal. The page beneath must be fully usable. - Don't auto-reshow on return visits. - Don't include unrelated CTAs. - Don't ship without the **Hear the voice** sample — the voice IS the product, and visitors should hear it before they commit a Live API session. ## 6c. Capabilities info button (persistent in header) Add a persistent `(i)` icon in the top-right of the header. Click → opens a modal/panel titled **"What powers this app"**. **Panel contents (in this order):** **Gemini capabilities used (the hero list):** - **Gemini 3.1 Flash Live (`gemini-3.1-flash-live-preview`)** — the live audio + video model that sees what your camera sees and speaks back in real time. Post-Google-I/O 2026 acoustic nuance means pitch and pace sound considered, not robotic. Audio + video sessions cap at 2 minutes per turn — the app resumes seamlessly when you cross that limit. - **Gemini 3.5 Flash** — the post-I/O default flagship. Powers the subject classifier (figures out whether you're doing maths, code, recipe, etc.), the session preamble assembly (sets up the right vocabulary for the live session), the safety classifier (flags electrical/gas/structural work), and the transcript post- processor (cleans the saved record). - **Gemini 3.5 Flash + grounded search** — suggests one follow-up problem at the end of each successful session, grounded in real textbook / online sources (citations surfaced). - **Nano Banana 2 (`gemini-3.1-flash-image`)** — generates the welcome and empty-state imagery (a phone in a stand on a kitchen table at golden hour with a notebook). - **Gemini 3.1 Flash TTS (`gemini-3.1-flash-tts-preview`)** — used once during build to record the 12-second voice sample on the welcome screen. Cached as a static MP3; not called per visit. - **Live API function calling** — three synchronous functions (`markHint`, `markArrival`, `requestStillFrame`) let the model coordinate with the client without breaking the conversational flow. - **Firebase Auth** — Google and Apple sign-in. - **Firestore** — stores your sessions, syncs across devices. - **Firebase Storage** — keeps the still frames captured during sessions. - **Cloud Run gateway** — a server function proxies the Live API websocket so your `GEMINI_API_KEY` never leaves the server. Post-I/O 2026 you get free deploy of the first 2 apps to Cloud Run. **The Socratic discipline (the rule that defines this app):** - The model never gives the answer outright. It asks the next question. Three hints per session, visible at the top as filled dots. When the budget is exhausted, the model defaults to encouragement and reflection rather than capitulating. **The safety contract:** - The still-frame safety classifier runs on every captured still. If it sees mains electrical, gas, load-bearing structural, medical wound, firearm, chemical handling, or child-safety territory, the app surfaces a not-a- substitute banner that stays visible for the rest of the session. The Live API model is instructed never to authorise such work — only to help the user think through what they see and what a qualified professional would check. **Backend services this app depends on:** - Auth: see section 4b - Database: see section 4b - Storage: see section 4b — REQUIRES manual enable in Firebase console; AIS Build does not auto-provision Storage today. - Email: see section 4b — magic-link transcript sharing requires the sender domain to be authorised in Firebase Auth. - Apple sign-in: optional, requires an Apple Developer account and Service-ID config. See section 4b. - Payments: see section 4b (not used in v1). - External APIs: see section 4b. **Environment variables you'll need to configure:** - `GEMINI_API_KEY` — your Google AI Studio API key - `FIREBASE_PROJECT_ID` — your Firebase project id - `FIREBASE_SERVICE_ACCOUNT` — service-account JSON (server-side only) **Cost + privacy notes:** - One short paragraph per cost-sensitive capability: Live API sessions are billed by streaming-second; a 4-minute session (2 segments) is approximately $0.18. The text calls (classifier + preamble + post-processor + safety) add ~$0.01/session. - One short paragraph on privacy: where the data lives (your Firebase project), how to delete it (Settings → "Delete this session forever" — gone in 60 seconds), what is never sent for training, what is never stored (raw video frames beyond the live session window). **Documentation links:** - AI Studio Build docs - Gemini API Live (`gemini-3.1-flash-live-preview`) docs - Gemini API multimodal image, structured output, grounded search docs - Firebase Auth, Firestore, Firebase Storage docs **Accessibility:** same standards as the onboarding modal — focus trap, `Esc`, ARIA, restored focus. **Behaviour:** - Always available — single click from anywhere in the app. - Tooltip on the `(i)` icon: "How this app is built". - Mobile: opens as a full-screen sheet that slides up. - Should be the most honest part of the app — never hand- wave service requirements; never say "AI" without naming the specific Gemini model and capability. ## 6d. Detailed cost breakdown (deployer reads this BEFORE shipping) - **Live API session (`gemini-3.1-flash-live-preview`)** — billed by streaming-second. As of 2026-06, ~$0.075/min of audio in, ~$0.30/min of video in, ~$0.30/min of audio out (approximate; check current pricing). A typical 4-min session (one resume) ≈ ~$0.18. A 2-min session (no resume) ≈ ~$0.09. **Plan for ~$0.10–$0.20 per session.** - **Subject classifier (Gemini 3.5 Flash, low thinking)** — ~10 frames + a short system instruction, ~3,000 input tokens and ~150 output tokens → ~$0.006 per session. - **Session preamble assembly (Gemini 3.5 Flash, low)** — ~500 input tokens + 300 output → ~$0.004 per session. - **Safety classifier (Gemini 3.5 Flash, low)** — runs on each `requestStillFrame()` capture. Typical session has 2–4 stills → ~$0.004 per still → ~$0.012 per session. - **Transcript post-processor (Gemini 3.5 Flash, low)** — ~8,000 input tokens (transcript + still descriptions) + 1,000 output → ~$0.02 per session. - **Next-session suggestion (Gemini 3.5 Flash + grounded)** — ~$0.005 per call, run once per successful session. - **Hero / empty-state image (Nano Banana 2)** — ~$0.03/image. Generated once per app install (cached client-side). - **TTS voice sample (Gemini 3.1 Flash TTS)** — recorded ONCE at build time, cached as static MP3 in `/public/audio-samples/welcome-12s.mp3`. Approximately $0.0015 to generate; zero cost per visitor. - **Expected per-session cost (typical):** ~$0.13–$0.23 Gemini API spend. - **Heavy user (1 session per day):** ~$5/month Gemini total. - **Still-frame storage:** Firebase Storage standard tier ~$0.026/GB/month. A 1080p JPEG ≈ ~400 KB; 50 stills/month ≈ ~20 MB ≈ ~$0.0005/month. - **Cloud Run gateway:** free tier covers the first 2 million requests/month and 360k vCPU-seconds. Live API proxying uses minimal CPU because frames pass through. Post-I/O 2026 free 2-app Cloud Run deploy covers this app out of the box. ## 7. Design language - **Mood:** A calm desk lamp on a study table at 11pm. The phone in a stand. A notebook open. A laptop screen showing code. A kitchen counter with a dough at the windowpane test. The carburettor housing open on a workshop bench. Not a tech-product showcase. Not a flashy AI demo. The moment between "I'm stuck" and "oh — I see". - **Typography:** Clean grotesque for app chrome and the live transcript (Inter or Geist). Display serif for session titles and the welcome headline (Source Serif Pro or Fraunces). A monospaced face for code-mode transcripts (JetBrains Mono or SF Mono). Body 16-17px, generous line-height in the transcript (1.7+) so the live caption is comfortable to read mid-session. - **Palette:** Warm parchment background `#F5F1EA` for the notebook surfaces and session library cards. Deep ink `#1B1714` for body. Tutor violet `#7C3AED` for the hint- budget dots, the active record button, and the moment-of- arrival highlight (matches the user's preferred Apple- doc accent). A soft tutor-violet tint `#EDE9FE` for the hint-budget background and the live-transcript pane. Tutor-pink `#DB2777` accent (sparingly, for the *I think I've got it* button to differentiate from the regular hint request). Highlight yellow `#CA8A04` reserved for the not-a-substitute safety banner with soft tint `#FEF9C3`. Tutor-green `#16A34A` only for the arrival confirmation dot. - **Imagery:** Photographic. A phone in a stand on a kitchen table at golden hour. A laptop screen showing code with a window-light reflection. A wooden workbench with the bicycle frame upside-down. Generated via Nano Banana 2 with prompts emphasising warm light, real imperfection (paper grain, fingerprint smudge on the phone screen), no people, no commercial branding. - **Hand-feel touches:** The hint-budget dots fade rather than vanish when used — the previous filled state is ghosted for half a second so the user sees what they spent. The live transcript pane has a soft scroll-shadow at the top and bottom so it reads like a continuous strip of paper passing through. The moment-of-arrival card has a thin watercolour-wash background, suggesting something hand-marked. The record button breathes (a 4-second sinusoidal opacity oscillation between 1.0 and 0.92) when the Live API session is connecting, settling to steady on connection. - **Spacing:** consistent 4-px base. Generous whitespace — the live transcript and the camera preview need air. - **Radius:** consistent token set (e.g. 8 / 14 / 22 px). Transcript turn cards use 8; the arrival card uses 14; the welcome card uses 22. - **Shadows:** subtle, warm-tinted. Avoid heavy drop- shadows. The hint-budget dots have no shadow. - **Motion:** purposeful — the record button's breathing oscillation while connecting; the hint dot's fade-on- spend; the moment-of-arrival's brief dim-around. Respect `prefers-reduced-motion`: replace the oscillation with a static colour change; replace the fade with an instant state change; replace the dim with a thin highlight. - **States:** every interactive element has hover, focus, active, disabled. Loading uses skeletons (the camera preview's "connecting…" placeholder is a soft pulse on a parchment rectangle, not a spinner). Empty states have domain-specific next-action labels ("Try the calculus practice problem", "Try the React-hook practice problem"). ## 8. Content generation rules - Write **realistic, specific copy**. NO Lorem Ipsum. NO generic placeholders. - Invent plausible names, problem statements, transcript snippets, and arrival-summaries that fit the domain. When inventing, lean on realistic patterns — `∫ x · sin(x²) dx`, a `useEffect` with a missing dependency, a windowpane test on sourdough, a VLOOKUP with the wrong column index, a wobbly Billy bookcase with loose cam-locks. - Tone: warm, direct, calm. This template is for someone who is stuck — they do not need cheerleading and they do not need to feel small. - Headlines: punchy and concrete. No 'Empower your X' filler. No 'AI-powered'. No 'Revolutionize your learning'. - Body copy: short paragraphs (2-4 sentences). Use lists where appropriate. - Plain language. Avoid jargon — except where the user already speaks it (the maths student wants to see "integral", "substitution"; the developer wants to see "useEffect", "dependency array"; the baker wants to see "windowpane test", "autolyse"). - Where the app outputs AI-generated content, never label it as "AI says" — let the tutor voice speak naturally. - The not-a-substitute disclaimer copy is the SAME wording every time, surfaced as a banner: "This isn't a substitute for an electrician / plumber / clinician / structural engineer. I can help you think through what you're seeing, but I'm not qualified to authorise the work." (Verb adjusts to the safety category.) ## 8a. Seed content (use these specific examples) Anchor every generated copy + sample data point in the content below. **Sample practice problems (built-in library):** - **Maths — Calculus by substitution.** Problem statement shown on screen: `∫ x · sin(x²) dx`. Pre-loaded camera view: a handwritten version on lined paper. `SessionPreamble`: - vocabulary_anchors: ["integral", "substitution", "u-substitution", "antiderivative", "differential", "du", "limits of integration"] - socratic_patterns: ["What would you pick as u?", "What does du equal?", "How does the integral transform?", "What does the new integral evaluate to?"] - safety_watchlist: [] - initial_greeting_one_line: "I can see your integral — walk me through what you've already tried." - **Code — React useEffect infinite loop.** Problem statement: a `useEffect` hook that includes a setter function in its dependency array, causing infinite re- renders. Pre-loaded camera view: a screenshot of VS Code with the problematic hook visible. `SessionPreamble`: - vocabulary_anchors: ["useEffect", "dependency array", "infinite loop", "stale closure", "render cycle", "state setter", "mount", "unmount"] - socratic_patterns: ["What's in your dependency array?", "What does each value in there control?", "What triggers a re-render here?", "What would you want that array to contain instead?"] - safety_watchlist: [] - initial_greeting_one_line: "I can see your useEffect in VS Code — walk me through what you're trying to do here." - **Recipe — Sourdough windowpane test.** Problem statement: "knead until the dough passes the windowpane test". Pre-loaded camera view: a ball of dough on a floured wooden counter. `SessionPreamble`: - vocabulary_anchors: ["windowpane test", "gluten", "autolyse", "bulk ferment", "stretch and fold", "elasticity", "translucent", "tear"] - socratic_patterns: ["What does the dough feel like right now?", "How does it stretch?", "What do you see when you hold it up to the light?", "Does it tear, or does it stretch thin?"] - safety_watchlist: [] - initial_greeting_one_line: "I can see your sourdough in progress — what does the dough feel like right now?" - **Spreadsheet — VLOOKUP #REF! error.** Problem statement: cell `D14` shows `#REF!` from a VLOOKUP formula. Pre- loaded camera view: a screenshot of a Google Sheet with the formula bar showing `=VLOOKUP(A14, $B$2:$E$50, 5, FALSE)`. `SessionPreamble`: - vocabulary_anchors: ["VLOOKUP", "lookup table", "column index", "exact match", "approximate match", "#REF!", "#N/A", "absolute reference"] - socratic_patterns: ["What's the third argument of your VLOOKUP?", "How many columns does your lookup table have?", "What range did you give VLOOKUP to search?"] - safety_watchlist: [] - initial_greeting_one_line: "I can see your #REF! in D14 — what does your VLOOKUP formula look like?" - **DIY — Wobbly Billy bookcase.** Problem statement: a Billy bookcase assembled but wobbly. Pre-loaded camera view: a photo of the back of a bookcase with cam-locks visible. `SessionPreamble`: - vocabulary_anchors: ["cam-lock", "dowel", "wooden peg", "Allen key", "flathead", "Phillips", "back panel", "carcass"] - socratic_patterns: ["Where do you feel the wobble?", "Are the cam-locks at the bottom snug?", "Is the back panel fully seated in its groove?"] - safety_watchlist: ["load_bearing"] - initial_greeting_one_line: "I can see your Billy bookcase — where do you feel the wobble?" - **DIY — Lawnmower won't start.** Problem statement: a petrol lawnmower pulls but won't fire. Pre-loaded camera view: a photo of a lawnmower with the air filter housing open. `SessionPreamble`: - vocabulary_anchors: ["spark plug", "air filter", "carburettor", "choke", "primer bulb", "fuel line", "compression", "flywheel key"] - socratic_patterns: ["When did you last change the spark plug?", "What does the air filter look like?", "Is fuel reaching the carburettor?", "What did the spark plug look like when you pulled it?"] - safety_watchlist: ["chemical_handling", "other_professional"] - initial_greeting_one_line: "I can see your mower — and I'll flag a heads-up that fuel and small engines are professional territory. When did you last change the spark plug?" - **Music — F-major barre chord buzzing.** Problem statement: an F-major barre chord on guitar with buzzing on the B and high E strings. Pre-loaded camera view: a photo of a guitar fretboard with a hand attempting the F chord. `SessionPreamble`: - vocabulary_anchors: ["barre", "fret", "fingerboard", "string", "buzz", "thumb position", "index finger", "knuckle"] - socratic_patterns: ["Which strings are buzzing?", "Where is your index finger pressing on those strings?", "What does your thumb look like behind the neck?", "How close to the fret is your pressure?"] - safety_watchlist: [] - initial_greeting_one_line: "I can see your F-chord attempt — which strings are buzzing?" - **Language — Spanish menu in Seville.** Problem statement: a menu item reads "merluza a la plancha con piquillos" and the user wants to understand it without pure translation. Pre-loaded camera view: a photo of a menu page. `SessionPreamble`: - vocabulary_anchors: ["merluza", "a la plancha", "piquillos", "white fish", "grilled", "Spanish cuisine", "cognate"] - socratic_patterns: ["What do you recognise here?", "Does 'plancha' remind you of any English word?", "What kind of food might 'merluza' be from the section it's in?"] - safety_watchlist: [] - initial_greeting_one_line: "I can see the menu — what do you already recognise?" **Sample arrival transcripts (the moment-of-arrival lines the model should produce):** - Calculus arrival: "That's it. You did the substitution; I just kept you company. Save the transcript?" - React arrival: "Nice catch on the dependency array. Type the empty brackets and watch your component settle." - Sourdough arrival: "That's the windowpane — you're there. The dough's ready to shape." - VLOOKUP arrival: "Four columns, asked for the fifth. You saw it. Change the five to a four and it'll resolve." - Billy bookcase arrival: "Two snug cam-locks, no more wobble. Want me to save the transcript?" - Lawnmower arrival: "Fresh plug, fresh start. Remember the banner — for anything fuel-system or carburettor-internal, a mechanic is the call." - F-chord arrival: "That's a clean F. Your index finger finally got the rolling pressure right behind the second fret." - Spanish menu arrival: "You've got it — grilled hake with piquillo peppers. Order with confidence." **Sample voice copy (UI strings):** - Welcome subhead: "Point your camera at what you're stuck on. Get a calm voice that asks the next question — never gives the answer." - Hear-the-voice button: "Hear the voice" - Practice-session CTA: "Try a practice problem" - Subject chips header: "What are you working on?" - Hint-budget tooltip: "Three small hints per session. I'll use them sparingly." - I'm-stuck button: "I'm stuck" - I-think-I've-got-it button: "I think I've got it" - End-session button: "End session" - Connecting state: "Opening the line — give me a second." - Reconnecting after 2-min cap: "Picking up where we left off." - Arrival card heading: "You did it." - Arrival card subhead: "Want to save the transcript?" - Save-transcript button: "Save transcript" - Not-a-substitute banner (electrical): "This isn't a substitute for an electrician. I can help you think through what you're seeing, but I'm not qualified to authorise the work." - Not-a-substitute banner (medical): "This isn't a substitute for a clinician. I can help you describe what you see, but I'm not qualified to diagnose or treat." - Low-confidence framing prompt: "Camera's a little dark — can you move closer or add light?" - 2-min warning: "We've got about 15 seconds left on this segment — I'll pick up right where we are." - Resume prompt: "Last time we got as far as picking u = x². Want to keep going from there?" - Family-mode redirect: "Let's get back to the problem on your paper." **Sample seed transcript turn-by-turn (for the demo session detail view):** - 0:00 — model: "I can see your integral — walk me through what you've already tried." - 0:08 — user: "I tried integration by parts but it got messy. I think substitution might work." - 0:14 — model: "Good instinct. What would you pick as u?" - 0:22 — user: "Probably x squared, since its derivative is x and there's an x in front." - 0:30 — model: "That's the choice I'd make. What's du?" - 0:36 — user: "du equals 2x dx." - 0:41 — model: "You've got 2x dx on the right and x dx in the integral — what does the integral become in terms of u and du, once you handle that factor of 2?" - 1:02 — user: "Half… half integral of sin u du?" - 1:08 — model: "That's it. Now finish." - 1:24 — user: "Negative one-half cosine u plus C, which is negative one-half cosine x squared plus C." - 1:32 — model: "Hold it up for me?" - 1:36 — user: [holds paper to camera] — model called requestStillFrame() at 1:37 - 1:42 — model: "That's right. You did the substitution; I just kept you company. Want me to save the transcript?" - 1:46 — model called markArrival("user solved ∫ x · sin(x²) dx by u-substitution, u = x², arrived at -1/2 cos(x²) + C", 0.95) ## 9. Media & assets - **Hero image (landing screen):** A photographed-looking shot of a phone in a stand on a kitchen table at golden hour, a notebook open showing a handwritten calculus problem, the phone screen showing the Live Tutor record view. Generate via Nano Banana 2 with a prompt emphasising "warm afternoon light through a kitchen window, no people in frame, a phone in a wooden stand showing a clean app interface, a paper notebook open beside it with a handwritten integral, soft shadow under the phone, real worn paper, a coffee mug just out of the frame". - **App icon / wordmark:** Set in the display serif. Slightly worn paper texture behind it. No icon — just type ("Live Tutor"). - **Empty-state illustration:** A simple line drawing of a phone in a stand pointing at a notebook with a question mark on the page. Hand-drawn aesthetic, not a flat icon. Generate once at build time via Nano Banana 2 (`gemini-3.1-flash-image`), 1:1 WebP at 1024×1024, prompt: "single hand-drawn ink line illustration of a smartphone in a small upright stand on the left tilted slightly toward an open paper notebook on the right with a single hand-drawn question mark on the visible page, off-white paper background, slight pen imperfection, no shading, no colour fill, no text, no commercial branding". Ship as a seed asset at `/public/samples/empty-state-phone-notebook.webp`. - **Practice-problem cover images:** One photographed- looking image per practice problem, generated via Nano Banana 2. The maths one shows the handwritten integral on paper; the code one shows a laptop screen with VS Code; the recipe one shows dough on a wooden counter; the bookcase one shows the back of a half-assembled Billy; etc. - **Subject-chip icons:** Hand-drawn icons (lucide-react with hand-tuned strokes) — a square root sign for maths, an angle bracket for code, a whisk for recipe, a small grid for spreadsheet, a hex key for DIY, a treble clef for music, a speech bubble for language. - **Cached voice sample audio:** Generated once via `gemini-3.1-flash-tts-preview`, served as a static MP3 from `/public/audio-samples/welcome-12s.mp3`. The sample is a 12-second Socratic exchange: model asks "What would you pick as u?", a beat of silence, user voice (also TTS) says "x squared", model says "That's the choice I'd make. What's du?". - **Stock fallbacks:** If Nano Banana image generation fails, fall back to the photographed sample image from `/public/samples/sample-desk.jpg` (3:2 WebP, 2048×1365 — ship as a seed asset; recreate via Nano Banana 2 (`gemini-3.1-flash-image`) with the prompt: "photographic phone in a wooden stand on a kitchen table at golden hour, paper notebook open beside it with a handwritten integral on the visible page, soft shadow under the phone, real worn paper, a coffee mug just out of the frame, no people, no commercial branding"). Never to a generic AI-generated abstract. - **Generated imagery:** prefer Nano Banana 2 over stock photography. Prompt for warmth, asymmetry, slight imperfection. Avoid glossy AI render look. - **Optimisation:** WebP/AVIF, `loading="lazy"`, explicit `width`/`height` to prevent layout shift. - **Icons:** `lucide-react` for UI. Use sparingly. ## 10. Interactivity & states - Every interactive element has hover, focus, active, and disabled states. - Forms validate inline and show specific error messages. "Camera permission denied — check your browser's site settings" is the right shape, not "Error". - Loading states use skeletons that match the eventual layout. The connecting state of the Live API session uses the breathing record button (a 4-second sinusoidal opacity oscillation), not a spinner. - Empty states explain the next action with domain- specific labels: "Try the calculus practice problem", "Try the React-hook practice problem". Never a generic "Get started". - Smooth scroll for in-page anchors. - The Live API session's transcript streams in real time — caption appears as the model speaks, slightly behind the audio. User turns appear when audio transcription is ready. - If the Live API connection fails, show a calm, specific error ("We couldn't open the live session — check your network and try again. Your session has not been charged.") and offer retry. - The hint-budget dots animate as they're spent: the dot fades from filled to outline over 300ms, with a faint ghost of the filled state lingering for 500ms. With `prefers-reduced-motion`: instant state change. - The not-a-substitute banner, when surfaced, slides in from the top of the transcript pane over 250ms and stays pinned for the rest of the session. With `prefers- reduced-motion`: appears instantly. - The arrival moment dims the screen around a centred card over 400ms; the card fades in over 200ms. With `prefers- reduced-motion`: a thin highlight ring around the card instead of the dim. - All AI-generated content respects token-by-token streaming where the Live API provides it; the caption pane updates as audio arrives. - The 2-minute video session warning surfaces at 1:45 as a slim non-blocking inline note ("about 15 seconds left on this segment — I'll pick up right where we are"). - Resume of a previously-capped session is one tap from the session library; the resume opens a new Live API connection with a fresh system message summarising where the previous segment left off (text-only summary, not a re-stream of frames). ## 11. Tech & responsive requirements - **Stack:** React + TypeScript + Tailwind CSS. Functional components + hooks. Use Shadcn UI primitives where appropriate. - **Build runtime:** AI Studio Build — full-stack with Cloud Run server-side functions. All Gemini API calls happen server-side; API key lives in Secrets Manager, never in client bundle. - **Live API gateway:** a Cloud Run server function opens the Live API websocket using the project's `GEMINI_API_KEY` and proxies frames + audio between the client and Google. The client receives short-lived ephemeral tokens (10-min TTL) bound to the user's Firebase Auth UID. **Do not** put `GEMINI_API_KEY` in the client bundle. - **Audio capture:** the Web Audio API (specifically `AudioWorklet`) for capture + downsample to 16kHz PCM mono. `getUserMedia` for the audio + video track. The client must request `audio` and `video` permissions separately and degrade gracefully if only audio is granted (audio-only mode allows 15-minute sessions). - **Audio playback:** Web Audio API for playback of the PCM 24kHz mono stream from the Live API; queueing must handle jitter without dropping audio (a 200ms playback buffer is a reasonable default). - **Video capture:** `getUserMedia` with `video: { width: 1280, height: 720 }` constraint; sample at 1 FPS to the Live API channel via `requestAnimationFrame`-throttled capture. - **Still-frame capture (non-blocking ring buffer):** the `requestStillFrame()` function call MUST NOT block the Live audio channel. The client maintains a 3-frame 1080p ring buffer continuously sampled from the video stream. When the Live API fires `requestStillFrame`, the client dispatches the latest cached frame asynchronously to an upload worker; the active audio/voice loop is never blocked, and the upload happens off the bidirectional gRPC/WebSocket stream. The worker uploads to Firebase Storage, server-side re-uploads to the Gemini Developer API Files API, and surfaces the resulting `files/*` resource name (passed via `fileData.fileUri`) back to the gateway so the model can reference it in the next turn. - **Live API session state recovery.** The Live session caps at 2 minutes (audio+video) / 15 minutes (audio-only); on every Live tick the client persists a `SessionSyncState` to `sessionStorage`, and on reconnect it passes a concise context summary block into the first system message of the next handshake cycle so the model picks up where it left off: ```typescript interface SessionSyncState { activeSessionId: string; accumulatedSegments: Array<{ speaker: string; text: string; timestamp: number }>; currentActiveRubricIndex: number; // current step in the Socratic plan activeStillUris: string[]; // files/* refs captured so far hintEventsSoFar: Array<{ index: number; size: 'small' | 'medium'; givenAtIso: string }>; } ``` - **Model selection:** explicitly pin `gemini-3.1-flash-live-preview` for the live session and `gemini-3.5-flash` for every text-only call. Set `thinkingLevel: "low"` on every 3.5 Flash call. Do NOT set `thinkingConfig` on the Live API, image-gen, or TTS calls. - **Database:** Firestore (auto-provisioned by AI Studio Build). Show the practice library on first launch. - **Auth:** Firebase Auth — Google sign-in by default; Apple sign-in next to it; magic-link email as fallback. - **Storage:** Firebase Storage for still frames. Pre- signed URLs only. - **Mobile-first.** Verify layouts at 375 px (iPhone SE), 768 px (iPad), 1024 px, 1440 px+. The active session screen MUST work at 375 px with the camera preview, the hint dots, the transcript pane, and the action buttons all visible without scrolling. - Use `clamp()` for fluid typography. Prefer container queries over media queries for component-level responsiveness. - Use `dvh` / `svh` instead of `vh`. Respect safe-area insets on iOS (the floating action buttons must clear the home indicator). - Zero horizontal overflow at any width. Zero layout shift on load. - Persist user data in Firestore. The active session is ephemeral until the user taps **Save transcript** — saved transcripts include the turn list, the still URIs, the hint events, the arrival event. - Wake-lock during an active session via the Screen Wake Lock API (the user should not have their screen sleep mid-Socratic-question); falls back gracefully if unsupported. ## 12. Accessibility (WCAG 2.2 AA) - Semantic HTML — `header`, `nav`, `main`, `section`, `article`, `footer`. - All interactive controls reachable by keyboard with a visible focus ring. - Color contrast ≥ 4.5:1 for body, 3:1 for large text and UI components. Tutor-violet `#7C3AED` on parchment `#F5F1EA` is verified ≥ 4.5:1. - All images have meaningful `alt` text. The practice- problem cover images use descriptive alt ("photograph of a handwritten integral ∫ x · sin(x²) dx on a paper notebook"). - Form fields have associated `