================ ONE-SHOT BUILD CONTRACT (read first) ================
Build this in Google AI Studio "Build" in ONE shot — a complete, working app,
no follow-up turns. These are hard rules, not suggestions:
1. TARGET = Full-Stack Web (Node server runtime, secrets, Firebase allowed).
If you target Native Android instead, you MUST drop all server/DB/Workspace/
secrets and persist locally (Room / SharedPreferences) only.
2. PIN EVERY MODEL CALL — never let the agent auto-select (it downgrades on quota):
- Reasoning / text -> gemini-3.5-flash (thinkingLevel: minimal|low|medium|high)
- 4K image + legible text -> gemini-3-pro-image (image_size "4K", up to 14 refs)
- High-volume image -> gemini-3.1-flash-image
- Expressive TTS -> gemini-3.1-flash-tts-preview (inline tags e.g. [whispers])
- Realtime audio/video (WebSocket) -> gemini-3.1-flash-live-preview
- Sandboxed agent -> antigravity-preview-05-2026
3. DIVISION OF LABOR — the model ONLY parses/extracts to a strict responseSchema.
ALL math, money (store currency as integer minor units / cents), sorting,
balancing and graph logic run in deterministic TypeScript/Python. The model
must never compute totals, splits or balances itself.
4. responseSchema sanitation — no regex patterns, no fixed-length tuples, no
format validators in the schema (they crash the OpenAPI engine). Enforce those
in server-side code AFTER parsing the JSON.
5. responseSchema and google_search grounding are MUTUALLY EXCLUSIVE in one call.
6. CODEGEN — split large output into modular, single-responsibility files so no
file is truncated by the output-token cap.
7. Every external call gets a graceful fallback (e.g. manual paste if a Workspace
read fails). Never a silent dead end.
8. ROBUST STORAGE & CANVAS — Wrap all `localStorage`/`sessionStorage` operations (especially JSON parsing and writes) in `try-catch` blocks to prevent crashes in private windows or quota overflows. Canvas drawing elements must dynamically handle window resize and scale pixel density (`window.devicePixelRatio`) to avoid blurry graphics on retina displays.
=====================================================================
# MUST OBEY — Mobile-first build requirements
This app's PRIMARY surface is a mobile phone. Build it impeccably on mobile FIRST, then verify on tablet and desktop. Treat the rules below as non-negotiable hard constraints, not suggestions.
## Viewports to verify (every screen, every state)
- 320 px, 360 px, 375 px, 390 px, 414 px, 480 px
- 768 px, 834 px (iPad portrait / Pro 11)
- 1024 px, 1280 px, 1440 px, 1920 px, 2560 px
- Plus: 200% browser zoom, landscape orientation on every mobile width, iPhone with safe-area insets visible
## Hard layout rules
- Mobile-first CSS. Default styles target mobile; `@media (min-width: ...)` for larger viewports.
- Use `dvh` and `svh` instead of `vh` for full-height surfaces (iOS Safari URL-bar bug).
- Use `clamp()` for fluid typography across all viewports.
- Prefer container queries (`@container`) over media queries for component-level responsiveness.
- Use `min(100%, ...)` widths so content never overflows. Zero horizontal overflow at any viewport.
- Add `` to every page.
- Apply `padding: max(safe-area-inset-X, fallback)` on every edge-bleeding container so notched iPhones in landscape never clip content.
- Wide tables and code blocks scroll INSIDE their container (`overflow-x: auto`), never push the body.
- Use `background-attachment: scroll` on mobile, not `fixed` (iOS Safari repaint bug).
- Avoid `backdrop-filter` on animated elements. Use it sparingly on static surfaces only.
- **Canvas Scaling**: Canvases must dynamically scale with window resize events and properly handle high-DPI screens (`window.devicePixelRatio`). Set physical dimensions (`canvas.width`/`canvas.height`) using pixel ratio and render relative to this grid, using CSS to control responsive viewport scaling.
- **Robust Storage**: Every access to `localStorage`/`sessionStorage` (especially `JSON.parse` of loaded state or writes) MUST be wrapped in a `try-catch` block to handle disabled storage, private browsing mode, quota limits, or corrupted JSON gracefully. Fall back to a robust in-memory object store.
## Touch & accessibility
- Tap targets ≥ 44 × 44 px on touch (Apple HIG). Increase to 48 px under `@media (hover: none) and (pointer: coarse)`.
- All interactive controls reachable by keyboard with a visible focus ring; respect `:focus-visible`.
- Color contrast ≥ 4.5:1 for body text, 3:1 for UI components.
- All images have meaningful `alt`. Decorative images use `alt=""`.
- Respect `prefers-reduced-motion: reduce` — zero animation durations under that query.
- Forms validate inline; error messages are specific, not "Invalid input".
- Modals: focus trap, `Esc` closes, `role="dialog"`, `aria-modal="true"`, focus restored on close.
## Performance bar (Lighthouse mobile, throttled 3G/4G)
- LCP < 2.5 s · INP < 200 ms · CLS < 0.1
- JS bundle gzip < 200 KB mobile-first; lazy-load non-critical screens via `React.lazy` / dynamic imports.
- No render-blocking resources above the fold.
- Images: WebP/AVIF preferred, `loading="lazy"`, explicit `width`/`height` attributes (zero CLS), `srcset` for retina.
- Videos: `preload="metadata"`, low-resolution poster, max 720p mobile fallback. Never autoplay with audio.
- Fonts: `font-display: swap`; preload only the one used above the fold.
- Smooth scroll honoured via CSS `scroll-behavior: smooth` with reduced-motion fallback.
## Pre-ship mobile checklist (the deployer MUST verify before declaring done)
1. Open at 375 px in DevTools — every screen scrolls vertically only; zero horizontal scroll.
2. Browser zoom 200% — layout reflows without overlap.
3. iPhone Safari with the URL bar visible AND landscape — no content under the home indicator; no notch clipping.
4. iPad portrait (768 px) and landscape (1024 px) — no awkward gaps; tablet-specific breakpoints land cleanly.
5. Tap every interactive element with a thumb at real-device size — every target is easy to hit.
6. `prefers-reduced-motion: reduce` — every transition / animation skips cleanly, scroll-behavior becomes instant.
7. Lighthouse mobile score ≥ 90 across all 4 categories.
8. Zero `console.error` and zero CLS shift in real-device testing on a mid-tier Android (e.g. Pixel 6a) and an iPhone SE.
---
The original template starts below. All rules above apply on TOP of whatever this template specifies.
---
# Allergen Menu
## 1. Project
**Allergen Menu** is a pocket translator for eating across languages
when a single ingredient can put you in the hospital. The user lifts
their phone over a printed menu in any script — Portuguese, Japanese,
Arabic, Thai, Vietnamese, Tamil, Bengali, Korean, Mandarin, Amharic,
Hindi, Urdu, Swahili, Khmer, Farsi, Greek — and the app reads the menu
as a chef would: every dish parsed, every named ingredient resolved
against the traveller's allergen profile, every cross-contact risk
flagged, every safe dish gently lit. The forbidden dishes are not
hidden; they are explained, and where a kitchen could plausibly swap
the offending ingredient, the app provides the exact phrase to say to
the waiter in the local language so the traveller can ask without
mime, panic, or the cousin-who-speaks-the-language on FaceTime.
This is the kind of app a shellfish-allergic traveller opens on her
second night in Lisbon, when she's already exhausted, the menu is
hand-chalked, and the word *amêijoas* might be clams or might be
the name of the chef's grandmother — because misreading it costs her
an epi-pen and a hospital night. It is also the kind of app a parent
opens on a family trip to Tokyo with their coeliac eight-year-old, in
a town where soy sauce hides wheat and *tempura* batter does too;
and the kind of app a peanut-allergic teenager opens in Marrakech
where ground peanut sneaks into tagine, into harira, into pastilla.
Same shape of moment, different city, different ingredient — the
diner cannot read the menu, cannot fully trust the waiter's English,
and cannot accept ambiguity.
The single demo that proves the magic: the user lifts the phone over a
Lisbon menu chalked in Portuguese. Within four seconds the camera
view becomes an overlay — the *bacalhau à brás* is a steady green
glow; *amêijoas à Bulhão Pato* is red with a calm red border; the
*polvo grelhado* is a dimmer orange ("octopus — flagged, you mark
this safe yourself"). Tapping the red dish doesn't hide it. It opens
a small panel: "This dish contains clams (*amêijoas*) and is finished
with garlic and white wine — shellfish risk is direct, not
cross-contact. Ask the waiter for the *bacalhau à brás* instead.
If you want this dish without shellfish: 'Desculpe — sou alérgica
a marisco. Tem outro prato sem amêijoas que recomende?' (the literal
English: 'Sorry — I'm allergic to shellfish. Do you have another
dish without clams that you'd recommend?')."
And in the harder cases — kitchens that fry everything in the same
oil, sauces with thirty undisclosed components, festival foods made
once a year by someone's grandmother — the app does not pretend to
know what it cannot know. When in doubt it goes red, not green. It
says, in the local language, *what to ask*, not *what's safe*. The
traveller still talks to the kitchen; the app gives them the words.
**Tagline:** _Eat safely in any city, any language, with every allergen, swap, and waiter phrase ready to read._
## 2. Target audience
- Adults with serious food allergies (shellfish, tree nuts, peanuts, sesame, dairy, egg, soy, gluten/wheat) travelling beyond the languages they read
- Parents of allergic children — the higher-stakes case, where misreading a menu means a four-year-old in a foreign A&E
- Coeliacs travelling in cuisines where gluten is a default condiment (Japan's soy sauce, China's vinegar, much of South Asia's asafoetida-with-wheat-binder)
- Diaspora returnees eating in their family's country who can speak the language but cannot read modern menus across scripts (a second-generation Tamil-American who speaks but does not read Tamil; a third-generation Filipino-Australian whose Tagalog stopped at hello)
- Religious-observance eaters who need clarity on pork, alcohol, beef, or gelatine across cuisines (halal, kosher, Jain, Hindu vegetarian, Buddhist)
- Multi-allergen households where one person has tree nuts, another has lactose, and one is fine with everything — the meal must work for the table
- People with oral-allergy syndrome whose triggers are seasonal and cross-react with specific fruits or birch-related foods
- Adults newly diagnosed with an allergy in midlife, still learning what to look for in unfamiliar cuisines
- Cabin crew, sailors, aid workers, and frequent business travellers crossing several food cultures in a week
- Allergists, school nurses, and travel-clinic staff who recommend a tool to patients before a trip
## 3. Core value propositions
Surface these clearly through copy, visual emphasis, and section ordering — they are the reasons users pick this app.
- **Reads any menu, in any script** — printed, chalked, handwritten, photographed crookedly across a tablecloth. Gemini 3.5 Flash's multimodal reading handles Portuguese, Spanish, French, Italian, German, Greek, Turkish, Arabic, Hebrew, Farsi, Urdu, Hindi (Devanagari), Bengali, Tamil, Punjabi (Gurmukhi or Shahmukhi), Sinhala, Khmer, Thai, Vietnamese (chữ Quốc ngữ), Tagalog, Mandarin and Cantonese (traditional or simplified), Korean (Hangul), Japanese (kanji + kana), Amharic (Ge'ez), Swahili — the same call, in one shot.
- **When in doubt, RED** — the app never green-lights an ambiguous ingredient. If a sauce on a Thai menu is described as "house special" with no parsed ingredient list, that dish is orange or red until the kitchen confirms. Safe dishes glow green only when every named ingredient is unambiguously outside the user's profile. The app is biased to caution, not coverage.
- **The waiter phrase, in their language** — for every red or orange dish, the app gives the exact Portuguese / Japanese / Arabic phrase to ask the kitchen, with a one-line literal back-translation so the traveller knows what they're saying before they say it. Phrases are written in the politeness register the cuisine expects, not literal translations from English.
- **Cross-contact, not just contains** — peanut-free pad thai cooked in the same wok as a peanut sauce is not safe for an anaphylactic peanut allergy. The app flags cross-contact risk separately from "contains" — same kitchen, same fryer, same wok, same cutting board — and pulls in the cuisine-specific defaults (one fryer for everything is the default in much of Tokyo izakaya; one wok in a small Sichuan kitchen).
- **The Live overlay reads as you point** — Gemini Live API streams the camera feed and updates the safe/unsafe overlay in near-real time, so the user can sweep across a long menu without queuing five photo captures. The Live overlay is the hero interaction. Photographs are the fallback for menus too dark or too cramped to scan live.
- **Profiles for everyone at the table** — a profile per diner; the menu colours can show "safe for all four of us", "safe for me and the kids", "safe for the kids only". Saved profiles travel between trips and across the family device.
- **Offline-first** — once a menu has been read, the parse is cached locally and works offline through the meal, even when the restaurant's Wi-Fi gives out and the user has no roaming data. The waiter phrase is pre-rendered, so it can be read aloud or shown to the waiter without a network.
- **It never replaces the conversation with the kitchen** — the app's role ends at the table; the kitchen's word is final. The app makes the conversation possible; it does not pretend to know what only a chef can know.
## 4. Features to build
- Live camera overlay (mobile-first) — sweep the phone across a menu and watch dishes light up green / orange / red in near-real time using the Gemini Live API
- Static photo capture for menus too dim, too small, or too far to read live — single shot, multi-shot for a multi-page laminated menu
- Upload from the photo library — for the user who already snapped the menu on the way back from the bathroom
- Multilingual menu parse — reads the menu in its source script, transliterates where helpful, translates to the user's language only as a secondary line below the source
- Per-dish ingredient extraction — every named ingredient resolved to a controlled vocabulary keyed to the user's allergen profile
- Cross-contact inference — cuisine-specific defaults (single-fryer izakaya, single-wok bao kitchen, communal pot in fondue, communal bread basket where the same hands serve every table)
- Allergen profile per diner — multiple profiles per device; the table colour-codes the menu against all profiles at once
- Per-cuisine knowledge cards — what hides where (gluten in Japanese soy sauce; sesame in Middle-Eastern bread; peanut in West African groundnut stew; dairy in Indian "vegetarian" naan; egg in fresh pasta; shellfish stock in many Vietnamese phở)
- Waiter phrase generator — for any red or orange dish, the exact local-language phrase to ask the kitchen, with a literal back-translation, plus a polite-register and an informal-register variant
- Phrase audio — Gemini TTS reads the waiter phrase aloud in the local language so the traveller can hear the pronunciation, then play it from the phone if they prefer
- Show-to-the-waiter card — a large-print local-language summary the user can hand over: "I am allergic to [shellfish]. Severe. I cannot eat dishes that contain or were prepared near it." Pre-rendered for offline use.
- Confidence chips on every dish — green / orange / red are visual; the underlying reasoning ("contains: clams; cross-contact: low; confidence: 0.94") is one tap away
- "Tell me about this dish" — the user taps any dish and gets a short, factual description of what's in it, not a marketing blurb
- Profile-aware swap suggestions — "this dish has shellfish; the kitchen could swap to chicken thighs; ask: '*com frango em vez de marisco?*'"
- Cuisine-specific picker phrases for buffets, street food, and tasting menus where the dishes change ("a tasting-menu phrase that asks the chef to substitute around an allergen without ruining the chef's intent")
- Travel pack — pre-loaded knowledge for a city before the trip ("Lisbon for shellfish allergy" downloads the local-name vocabulary, common hidden sources, and the Portuguese waiter phrases so the app works air-side and on the metro)
- Trusted-restaurants list — restaurants that handled the allergy well are starred; the list syncs across the family devices
- Allergy translation card export — a PDF the user can print before the trip with the local-language phrase, an emergency phrase, and the local emergency number
- Family-table mode — at a table of four with three different profiles, the menu shows a small per-profile dot pattern next to each dish so each diner sees instantly what they can eat
- Symptom log — if something goes wrong, log it; the next entry to a similar restaurant warns the user. Private, on-device.
## 4b. Required Gemini capabilities + backend services
**This template's intelligence comes from the Gemini capabilities below. Wire them up explicitly — don't substitute generic LLM calls.**
### Gemini capabilities (the load-bearing intelligence)
- **Gemini Live API (multimodal video + audio)** — the hero. The camera feed streams to Gemini 3.5 Flash via the Live API; the model reads the menu as the user sweeps the phone across it and emits a stream of `DishVerdict` events that update the overlay in near-real time. The Live API handles the multilingual reading, the structured extraction, and the safe / unsafe verdict in one continuous session. Latency target: first verdict back within 1.5 seconds of a dish entering frame.
- **Multimodal image input (Gemini 3.5 Flash)** — fallback path. A still photograph of a menu is sent as `inlineData` (base64) or via the Files API (`files/*` resource name) and parsed as a single multi-dish call. Used when the user prefers a single capture over the Live sweep, or when bandwidth is too low for a Live session.
- **Structured output / JSON Schema** — every Live event and every static parse matches the `Menu` and `DishVerdict` schemas below. The schemas are seeded verbatim in each system instruction and as `responseSchema` (except where grounding is enabled — see below).
- **Multilingual reading (built into Gemini 3.5 Flash)** — handles Portuguese, Spanish, French, Italian, German, Greek, Turkish, Arabic, Hebrew, Farsi, Urdu, Hindi (Devanagari), Bengali, Tamil, Punjabi (Gurmukhi or Shahmukhi), Sinhala, Khmer, Thai, Vietnamese, Tagalog, Mandarin (traditional or simplified), Cantonese (traditional or simplified), Korean (Hangul), Japanese (kanji + kana), Amharic (Ge'ez), Swahili — in one call, switching scripts as needed.
- **Search grounding** — for the per-dish cuisine knowledge ("does *amêijoas à Bulhão Pato* typically contain dairy in Lisbon?"). Grounded calls return the canonical answer plus citations. The traveller can see *where* the answer came from before they trust it. **Important**: when a call uses `google_search` grounding, do NOT also set `responseSchema` — they are mutually exclusive in a single Gemini call. Instruct the model to emit JSON in the text body and parse server-side; read citations from `response.groundingMetadata.groundingChunks[].web.uri`.
- **Gemini TTS** (`gemini-3.1-flash-tts-preview`) — speaks the waiter phrase aloud in the local language so the traveller can hear pronunciation. Voice locale is picked via `languageCode` matching the menu's source language. Gemini 2.5 TTS does NOT support SSML `` or `` tags; pace via `…` and a blank-line `—` in the input text. A one-sentence style directive ("Speak warmly, slowly, with the politeness register a guest would use") is prepended to the input.
- **Thinking levels** — `medium` for the static menu parse and for the cuisine-grounding calls. `low` for individual `DishVerdict` events emitted from the Live session (latency matters more than depth on each frame; the model has the user's profile in its session context). The Live session is configured with the allergen profile up-front so each event is small and fast.
### Backend services
- **Auth — Required.** Firebase Auth with Google sign-in (auto-provisioned by AI Studio Build). **Apple sign-in is optional but user-configured** — it requires an Apple Developer account, Service ID, Key ID, and private key wired into Firebase Auth. **Magic-link email** (used to share a profile across a family's devices) requires the sender domain to be authorised in Firebase Auth.
- **Database — Required.** Firestore for `users`, `profiles`, `menus_cached`, `trusted_restaurants`, `symptom_logs`, `travel_packs`. Profiles are private to the owner; family-mode shares are explicit and revocable.
- **File storage — Required.** Firebase Storage for the photographed menu images (kept at upload resolution so the user can re-read them if a parse looks off). **Firebase Storage is NOT auto-provisioned by AIS Build today** — enable it in the Firebase console and wire the bucket name in before first capture. Pre-signed URLs only; menus are never publicly addressable.
- **Email — Required (transactional).** Magic-link family-mode share; the printable allergy translation card delivered as a PDF attachment if the user prefers email over device download.
- **Payments — Not needed for v1.** Free for personal use. A future "city pack subscription" could pre-load richer cuisine knowledge for paying users; v1 ships with a curated set of free city packs.
- **External APIs:** Gemini API for all intelligence. Optional: the local emergency number per country, sourced from a static JSON shipped with the app, not from a third-party API.
**Environment variables:** every secret (Gemini API key, Firebase service-account JSON) lives in environment variables — never in client bundle. Include a `.env.example`.
**Auth + data privacy reminders:** never log secrets · never store passwords in plain text · use HTTPS everywhere · honour 'delete my account' inside the UI · explicit opt-in for any analytics · medical data (allergen profile, symptom logs) is private to the owner and explicitly-invited family members · this app uses the Gemini API on the paid tier, where Google does not use your content for model training, per the Gemini API Additional Terms · symptom logs are stored encrypted at rest with a per-user key the user can rotate.
**Read this first — prompt-craft rules that apply to every call in this template:**
1. **Name the model variant explicitly** in every Gemini API call. Do not let the agent pick the model. See the per-call matrix below.
2. **Pin `thinkingLevel` explicitly** per call. See the matrix.
3. **Seed the JSON Schema as a fenced TypeScript / Zod block** in the system instruction or `responseSchema` field. The literal schema is below. **Convert the Zod schema to Gemini's `Schema` type via the SDK helper** before passing to `responseSchema` — do NOT pass raw Zod. **Numeric `min`/`max` constraints are documentation only inside `responseSchema`; clamp on the server after the response arrives.**
4. **Pin the system instruction separately** from user input. Use the `systemInstruction` field for persona + behavioural rules; use `contents` for user input. Never concatenate.
5. **Pre-declare tools as an enable/disable list** per call. The matrix below names which tools are enabled per call. Tools NOT listed for a call should be disabled.
6. **State negative constraints explicitly** — they are listed below. They are NOT "be careful" suggestions; they are hard rules the model must follow.
7. **Grounded responses can wrap JSON in ```json fences or add prose preamble.** Server-side, strip fences and brace-extract:
```typescript
function safeExtractJSON(raw: string): T {
const clean = raw.replace(/```json\s*|```/gi, '').trim();
const s = clean.indexOf('{'); const e = clean.lastIndexOf('}');
if (s === -1 || e === -1) throw new Error('No JSON boundaries in grounded response');
return JSON.parse(clean.slice(s, e + 1)) as T;
}
```
8. **Strip unsupported Zod modifiers before passing to `responseSchema`** — Gemini's OpenAPI subset rejects `.regex()` / `pattern`, fixed-length `z.tuple()`, and other custom validators. Use a sanitizer that flattens tuples to length-2 arrays and removes regex patterns before serializing. Validate those constraints in middleware AFTER parsing.
### Per-call model + tools matrix
| Call | Model | thinkingLevel | Tools enabled |
|------|-------|---------------|---------------|
| Live overlay session — reads menu video, emits `DishVerdict` events | `gemini-3.5-flash` (Live API) | low | (none) |
| Static menu parse → `Menu` schema (fallback path) | `gemini-3.5-flash` | medium | (none) |
| Per-dish cuisine knowledge (does this normally contain X?) | `gemini-3.5-flash` | medium | `google_search` grounding (no `responseSchema` on this call — see note) |
| Generate waiter phrase in local language with literal back-translation | `gemini-3.5-flash` | low | (none) |
| Generate "tell me about this dish" descriptive panel | `gemini-3.5-flash` | low | (none) |
| TTS speak the waiter phrase aloud in the local language | `gemini-3.1-flash-tts-preview` | n/a | n/a |
| Cross-contact assessment for cuisine + dish + allergen profile | `gemini-3.5-flash` | medium | (none) |
*Note for builders:* on TTS calls, omit `thinkingConfig` entirely — the field is not supported on that model. The `n/a` cells in this matrix are documentation only; do not serialise them into the request body. On the grounded cuisine-knowledge call, `responseSchema` is omitted (it is mutually exclusive with `google_search` in one Gemini call); the model emits JSON in the text body and the server parses it. Grounding citations come from `response.groundingMetadata.groundingChunks[].web.uri`, never from inside the JSON body.
### Primary structured-output schema (seed this verbatim in the prompt)
```typescript
import { z } from "zod";
const Allergen = z.enum([
"peanut",
"tree_nut",
"shellfish_crustacean", // shrimp, lobster, crab, crayfish
"shellfish_mollusc", // clam, oyster, mussel, scallop, squid, octopus
"fish",
"egg",
"milk_dairy",
"soy",
"wheat_gluten",
"sesame",
"mustard",
"celery",
"lupin",
"sulphite",
"pork",
"beef",
"alcohol",
"gelatine",
"msg",
"other",
]);
const Severity = z.enum([
"anaphylaxis",
"severe",
"moderate",
"intolerance",
"preference",
]);
const Profile = z.object({
profile_id: z.string(),
display_name: z.string(), // "Marta", "Kai (7)"
allergens: z.array(z.object({
allergen: Allergen,
severity: Severity,
notes: z.string().nullable(), // "epi-pen carried"
})),
cross_contact_tolerance: z.enum([
"zero", // anaphylactic — same fryer is unsafe
"low", // coeliac — same prep surface unsafe
"moderate", // mild — same kitchen ok if clean
"high", // intolerance only
]),
preferred_language: z.string(), // BCP-47, "en-GB"
});
const DishIngredient = z.object({
ingredient_verbatim: z.string(), // "amêijoas", "soy sauce"
ingredient_canonical: z.string().nullable(), // "clams", "soy sauce"
resolved_allergens: z.array(Allergen),
resolution_confidence: z.number().min(0).max(1),
presence: z.enum([
"named", // explicitly listed on menu
"implied_by_dish_name", // "pad thai" implies peanut
"implied_by_cuisine_default", // tempura implies wheat batter
"asked_kitchen", // user confirmed with waiter
"unknown",
]),
});
const CrossContactRisk = z.object({
source: z.enum([
"shared_fryer",
"shared_wok",
"shared_grill",
"shared_prep_surface",
"shared_utensil",
"shared_serving_vessel",
"shared_bread_basket",
"shared_dipping_sauce",
"kitchen_default_unstated",
]),
affected_allergens: z.array(Allergen),
cuisine_specific_note: z.string(), // "single fryer is the izakaya default"
severity_for_profile: z.enum(["high", "medium", "low"]),
});
const WaiterPhrase = z.object({
phrase_local_language: z.string(), // verbatim local script
phrase_local_transliteration: z.string().nullable(), // for non-Latin scripts
phrase_literal_back_translation: z.string(), // word-for-word English
phrase_polite_register: z.boolean(), // true = polite form (vous, anata, usted)
pronunciation_guide_ipa: z.string().nullable(), // best-effort IPA, never via SSML phoneme
language_code: z.string(), // BCP-47, "pt-PT"
});
const DishVerdict = z.object({
dish_id: z.string(),
dish_name_verbatim: z.string(), // "Amêijoas à Bulhão Pato"
dish_name_user_language: z.string().nullable(), // "Clams in garlic, white wine & coriander"
bounding_box_normalised: z.object({ // 0..1 in image coords
x: z.number().min(0).max(1),
y: z.number().min(0).max(1),
w: z.number().min(0).max(1),
h: z.number().min(0).max(1),
}).nullable(),
ingredients: z.array(DishIngredient),
cross_contact_risks: z.array(CrossContactRisk),
verdict: z.enum([
"safe_for_profile", // green
"ambiguous_ask_kitchen", // orange
"unsafe_for_profile", // red
"unknown_menu_unclear", // red by default — when-in-doubt rule
]),
verdict_explanation: z.string(), // one-sentence reason
waiter_phrase_if_swap_possible: WaiterPhrase.nullable(),
swap_suggestion_dish: z.string().nullable(), // "bacalhau à brás"
swap_suggestion_explanation: z.string().nullable(),
flagged_for_profile_ids: z.array(z.string()), // which diner's profile is affected
reading_confidence: z.number().min(0).max(1),
});
const Menu = z.object({
menu_id: z.string(),
source_language: z.string(), // BCP-47, "pt-PT"
cuisine_inferred: z.string(), // "Portuguese coastal"
restaurant_name_verbatim: z.string().nullable(),
city_inferred: z.string().nullable(), // "Lisbon" — null if no clue on menu
sections: z.array(z.object({
section_name_verbatim: z.string(), // "Entradas", "前菜"
section_name_user_language: z.string().nullable(),
dish_verdicts: z.array(DishVerdict),
})),
global_cross_contact_notes: z.array(z.string()), // "kitchen serves all-fried items from one fryer"
emergency_phrase: WaiterPhrase, // "I am having an allergic reaction — call an ambulance"
local_emergency_number: z.string().nullable(), // "112", "911", "119"
parse_confidence: z.number().min(0).max(1),
flagged_for_user_review: z.array(z.object({
field_path: z.string(),
reason: z.string(),
})),
});
type Menu = z.infer;
type DishVerdict = z.infer;
```
### Common failure modes (and how to avoid them)
- Agent silently downgrades `thinkingLevel` for the live menu reading to save quota — pin `gemini-3.5-flash` on the Live API session explicitly. Flash mis-reads similar dish names across scripts and silently green-lights ambiguous ingredients.
- Model green-lights a dish where one ingredient is unclear — the "when in doubt, RED" rule is broken. Fix: pin in the system instruction that any `resolution_confidence < 0.85` on an allergen-relevant ingredient forces `verdict = "ambiguous_ask_kitchen"` at minimum. The server must clamp this even if the model misses it.
- Cross-contact ignored — the dish is safe by ingredients but the kitchen shares a fryer or wok. Fix: cross-contact assessment runs as a separate pass with the cuisine default; the verdict is the worse of (ingredient verdict, cross-contact verdict).
- Waiter phrase is too polite or too informal for the register the cuisine expects — Japanese keigo for a casual ramen counter is awkward; tu form to an unknown waiter in Portugal is rude. Pin the polite default and provide an informal variant only when asked.
- Waiter phrase translates the allergen wrong — "shellfish" in Portuguese splits into *marisco* (broad) vs *crustáceo* (crustaceans only). Pin the canonical local-language allergen vocabulary per cuisine.
- TTS reads the local-language phrase with English phonetics — wrong. Pin TTS voice `languageCode` to match the menu's source language; Gemini 2.5 TTS pronunciation follows the voice's native locale. SSML `` is NOT supported on this model — do not try to override pronunciation that way.
- Grounded search and `responseSchema` set on the same call — fails. Fix: on the grounded cuisine-knowledge call, do NOT set `responseSchema`; instruct the model to emit JSON in the text body; parse server-side. Grounding citations live on `response.groundingMetadata.groundingChunks[].web.uri`, not inside the JSON body.
- Dish translated as a chef's blurb instead of an ingredient description — the "tell me about this dish" panel becomes marketing copy. Fix: pin the prompt to a factual ingredient-first description; ban marketing adjectives.
- Long context blown by sending every menu in the user's history to the Live session — keep the Live session scoped to the current menu plus the active profile only. Archive menus are loaded on demand.
- Verdict swings between green and red as the user sweeps the camera — the Live overlay flickers. Fix: server-side debouncing — a verdict only updates after two consistent events 400 ms apart, and a red verdict is sticky (red → green requires 3 consistent green events, never the other way).
- Symptom log shared without consent — fix: symptom logs are never shared in family-mode; only profiles are. The family-mode share screen says this in plain English.
### Negative constraints (hard rules)
- Do NOT green-light a dish with any ambiguous ingredient. If you cannot resolve an ingredient to a canonical name with confidence ≥ 0.85, the verdict is `ambiguous_ask_kitchen` at minimum. Bias to RED, never to green.
- Do NOT translate dish names in the user's language *instead of* showing the source. Always show the source-language dish name verbatim; the user-language gloss is a secondary line.
- Do NOT replace the waiter phrase with an English approximation. The phrase must be in the menu's source language. If you cannot write it in that language with confidence, omit the phrase and surface "ask in English / point at the dish" guidance instead.
- Do NOT mark cross-contact "low" by default. Cross-contact severity follows the user's `cross_contact_tolerance`. For an anaphylactic peanut allergy, single-fryer cuisines default to `high`.
- Do NOT invent ingredients to fill an empty list. If the menu prints only the dish name with no description, the ingredients array contains only those implied by the dish name (and they are marked `implied_by_dish_name` or `implied_by_cuisine_default`), and the verdict tilts orange or red.
- Do NOT recommend a dish you have not parsed. The swap suggestion must be another dish that is *on this menu* and that you parsed with `verdict: safe_for_profile`. Do not suggest "have the pad thai" if pad thai is not on the menu.
- Do NOT modernise the menu's spelling. *Bacalhau à brás* keeps its accent; *raştafonduğu* keeps its diacritics; *香港粥* keeps its traditional characters even if a simplified equivalent exists.
- Do NOT speak medical advice. The app names ingredients and risks; the diagnosis is the user's allergist's, not the app's. The waiter phrase is "I am allergic to X", never "you cannot have X".
- Do NOT use the user's allergen profile, symptom log, or trusted-restaurants list to train or fine-tune any model. Use the Gemini API on the paid tier, where Google does not use your content for model training, per the Gemini API Additional Terms.
- Do NOT auto-share trusted-restaurants or symptom logs in family-mode. Only profiles are shared; everything else is per-user.
- Do NOT claim a kitchen is "allergy-safe" — the app never blesses a kitchen. The user does, by starring it after a meal.
### Per-call `systemInstruction` strings
Use these as the literal `systemInstruction` field for each Gemini API call the built app makes. They complement the series-wide rules already uploaded as the global instructions file (`00-series-instructions.txt`).
### Call: Live overlay session — reads menu video, emits `DishVerdict` events
Model: `gemini-3.5-flash` (Live API) · thinkingLevel: low · Tools: (none)
```
You are reading a printed, chalked, or handwritten restaurant menu
through the user's camera feed in real time. The user is a traveller
with one or more food allergies. They are pointing the phone at the
menu in a restaurant. Your job is to identify every dish that enters
frame and emit a DishVerdict for each, as a stream of small structured
events.
The active allergen profile is in your session context. It contains
the diner's allergens, severities, cross-contact tolerance, and
preferred language. Treat it as the only truth about what is unsafe.
Languages you will encounter on menus include but are not limited to:
Portuguese, Spanish, French, Italian, German, Greek, Turkish, Arabic,
Hebrew, Farsi, Urdu, Hindi (Devanagari), Bengali, Tamil, Punjabi
(Gurmukhi or Shahmukhi), Sinhala, Khmer, Thai, Vietnamese (chữ Quốc
ngữ), Tagalog, Mandarin (traditional or simplified), Cantonese
(traditional or simplified), Korean (Hangul), Japanese (kanji +
kana), Amharic (Ge'ez), Swahili. Read the menu in its source script.
For each dish you can resolve to a name and a rough ingredient list:
- emit a DishVerdict matching the provided schema
- include a normalised bounding box (0..1 in the frame's coordinate
system) so the client can overlay the verdict over the dish
- include verdict: safe_for_profile (green), ambiguous_ask_kitchen
(orange), unsafe_for_profile (red), or unknown_menu_unclear (red)
- include a one-sentence verdict_explanation that names the specific
ingredient or risk
Hard rules:
- When in doubt, RED. Never green-light an ambiguous ingredient. If
resolution_confidence on any allergen-relevant ingredient is below
0.85, the verdict is ambiguous_ask_kitchen at minimum.
- Read the menu's source-language script verbatim. Do not modernise
spelling or transliterate unless the user's preferred_language is
different from the menu language, in which case provide
dish_name_user_language as a secondary field.
- For dishes implied by cuisine default (tempura → wheat batter,
pad thai → peanut, tagine → ground peanut in some regions, naan →
dairy), mark the ingredient as implied_by_cuisine_default and let
the verdict reflect the implication.
- Cross-contact is part of the verdict. If the cuisine default is a
single fryer (much of Tokyo izakaya), a single wok (small Sichuan
or Cantonese kitchens), a shared grill (Korean BBQ), a shared
bread basket (much of European bistro), include cross-contact
risks in the DishVerdict and reflect them in the verdict.
- Do NOT invent ingredients. If the menu prints only the dish name,
the ingredient list contains only what the dish name implies, and
the verdict tilts orange unless the dish is obviously safe.
- Do NOT recommend a dish that is not on the menu in this session.
swap_suggestion_dish must be another dish you have parsed in this
session with verdict: safe_for_profile.
- Sticky red. Once you emit a red verdict for a dish, do not flip it
to green in a later event unless the user has explicitly tapped
"I asked the kitchen and they confirmed" — the client will signal
that as a session event.
- waiter_phrase_if_swap_possible is generated for red and orange
dishes where a swap is plausible. The phrase is in the menu's
source language, in the polite register the cuisine expects.
emit DishVerdict events as a stream. Do not buffer the whole menu and
send one big object. Latency matters; the user is sweeping the
camera.
No commentary outside the structured events.
```
---
### Call: Static menu parse → `Menu` schema (fallback path)
Model: `gemini-3.5-flash` · thinkingLevel: medium · Tools: (none)
```
You receive one or more photographs of a restaurant menu. The user is
a traveller with one or more food allergies. The active allergen
profile is in the request context.
Read the menu in its source language. Extract every section, every
dish, and every named ingredient. Resolve ingredients to canonical
names. Emit verdicts (safe_for_profile / ambiguous_ask_kitchen /
unsafe_for_profile / unknown_menu_unclear) per dish, matching the
Menu schema.
Submit multipage menus as a SINGLE call with multiple images in order.
Upload each page via the Gemini Files API (`files/*` resource name) or send as
`inlineData` (base64). Do NOT pass Firebase Storage public URLs
directly to `generateContent` — the API does not fetch them.
Hard rules:
- When in doubt, RED. Same rule as the Live session. Any allergen-
relevant ingredient resolved with confidence < 0.85 forces
ambiguous_ask_kitchen at minimum. The server will clamp this even
if you miss it.
- Preserve every diacritic exactly. Portuguese ã, ç, ô; Spanish ñ;
French è, ç, à; Turkish ş, ğ, ı; Vietnamese ă, ơ, ư with tone
marks; Polish ł, ą, ę; German ü, ö, ä, ß; Tagalog ñ; Arabic and
Hebrew base+vowel forms when present. Render in the exact Unicode
character.
- Identify the cuisine. cuisine_inferred guides the cross-contact
defaults — single-fryer izakaya, single-wok small Cantonese,
communal bread basket bistro, etc. Pick the closest match; if you
cannot tell, leave it null and the server will default to
conservative cross-contact.
- Do NOT translate dish names in place of the source. Always include
dish_name_verbatim; dish_name_user_language is secondary.
- Do NOT extrapolate to nutrition advice. The app is about allergens,
not calories, not macros, not "is this healthy".
- waiter_phrase_if_swap_possible is generated for red and orange
dishes where a swap is plausible — i.e., the offending ingredient
is a swappable accent (the shrimp on a pasta), not a structural
component (the cheese in a four-cheese pizza).
- flagged_for_user_review names any dish where confidence is below
0.7 with a one-sentence reason.
- global_cross_contact_notes captures kitchen-wide risks visible from
the menu: "menu mentions a single fryer", "kitchen serves communal
bread", "tasting menu — chef's discretion on substitutions".
- emergency_phrase is generated in the menu's source language. It
says, in the politeness register of the cuisine: "I am having an
allergic reaction. Please call an ambulance." plus a literal
back-translation in the user's preferred language.
Output ONLY the Menu JSON matching the provided schema. No commentary.
```
---
### Call: Per-dish cuisine knowledge (does this normally contain X?)
Model: `gemini-3.5-flash` · thinkingLevel: medium · Tools: `google_search` grounding
```
You answer a single, focused question: does this specific named dish,
in this specific cuisine, in this specific city, typically contain a
specific allergen? Use grounded search to verify against canonical
culinary references — restaurant association guides, regional cuisine
encyclopaedias, allergen advisory bodies in the relevant country.
Examples of questions you receive:
- "Does pad thai in Bangkok typically contain peanuts?"
- "Does naan in Mumbai typically contain dairy?"
- "Does tempura batter in Tokyo typically contain wheat?"
- "Does pastilla in Marrakech typically contain almond?"
- "Does kibbeh in Beirut typically contain bulgur (wheat)?"
- "Does feijoada in Lisbon typically contain pork?"
Hard rules:
- Use `google_search` grounding for every answer. Do NOT answer from
your own training data alone — cuisines evolve, regions vary, and
household variants differ from restaurant defaults.
- Distinguish "typically contains" from "commonly served with" — the
peanut in pad thai is structural; the peanut sauce alongside is a
condiment that can be omitted.
- Distinguish "in this city" from "in this country" — Lisbon
shellfish defaults differ from Porto; Tokyo izakaya defaults differ
from Kyoto kaiseki.
- Where regional variation is significant, say so and name the
variants ("in Bangkok street pad thai, peanut is structural; in
many Western adaptations, peanut is on the side").
- Output the response as JSON in the text body — NOT via
`responseSchema`. responseSchema and `google_search` cannot be set
on the same Gemini call.
- The server reads citation URLs from `response.groundingMetadata
.groundingChunks[].web.uri`. Do NOT include URLs in the JSON body;
you will hallucinate them.
JSON shape:
{
"dish_name_verbatim": "...",
"cuisine": "...",
"city_or_region": "...",
"allergen_asked": "peanut",
"typically_contains": true | false | "regional_variation",
"structural_or_condiment": "structural" | "condiment" | "garnish" | "n/a",
"kitchen_can_typically_omit": true | false,
"regional_variation_note": "..." | null,
"summary": "one sentence the diner can act on"
}
No commentary outside the JSON.
```
---
### Call: Generate waiter phrase in local language with literal back-translation
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none)
```
You generate a single phrase the traveller can read aloud or show to
the waiter, in the menu's source language, with a literal back-
translation in the user's preferred language.
Inputs you receive:
- the menu's source language (BCP-47)
- the dish in question (verbatim from the menu)
- the allergen at stake (canonical name)
- the severity (anaphylaxis / severe / moderate / intolerance / preference)
- the kind of phrase requested: "swap_request", "remove_request",
"check_kitchen", "tell_chef_severity", "emergency"
- the politeness register requested (polite default; informal optional)
Output a WaiterPhrase object matching the schema.
Hard rules:
- The phrase is in the menu's source language, written in the script
the waiter actually reads — never an English approximation.
- Polite register by default. Vous in French, the polite verb endings
in Korean, ます-form in Japanese, usted in Spanish (where regional
norms permit), -mu form in Turkish, the polite copula in Arabic.
An informal variant is generated only when asked.
- The phrase names the allergen using the canonical local-language
term, not a calque from English. "Shellfish" in Portuguese is
*marisco* (broad) or *crustáceo* (crustaceans only); pick based on
the allergen subtype. "Gluten" in Italian is *glutine*; "milk" in
Mandarin is 牛奶 (cow's milk specifically) or 乳製品 (dairy more
broadly).
- The literal_back_translation is word-for-word, awkward English on
purpose — the goal is to let the user verify the phrase says what
they think it says, not to read smoothly.
- For non-Latin scripts, include a transliteration in
phrase_local_transliteration so the user can pronounce it. Use the
most widely-taught transliteration system per script (Hepburn for
Japanese, Pinyin for Mandarin, IAST for Devanagari, etc.).
- The pronunciation_guide_ipa is best-effort. It is NOT a TTS
override — Gemini 2.5 TTS does not accept SSML tags. The
IPA is for the user to read.
- For severity: severe and anaphylaxis prepend a one-clause statement
of severity ("Sou alérgica grave a marisco — entro em choque") so
the waiter understands this is not preference.
- For emergency phrases, include a request to call the ambulance
using the local emergency number — but the number itself is
surfaced by the client, not by you.
Output ONLY the WaiterPhrase JSON. No commentary.
```
---
### Call: Generate "tell me about this dish" descriptive panel
Model: `gemini-3.5-flash` · thinkingLevel: low · Tools: (none)
```
You describe a single dish to a traveller who does not know the
cuisine. The description is factual, ingredient-first, and three
sentences at most.
Hard rules:
- Lead with the structural ingredients ("clams cooked in white wine,
garlic, coriander, and olive oil"). Not with adjectives ("a
beloved Portuguese classic bursting with flavour"). Ban marketing
adjectives outright.
- If a key ingredient is allergen-relevant, name it specifically by
its canonical English name AND the local-language name in
parentheses ("clams (*amêijoas*)"; "ground peanuts (*kacang*)").
- Describe how it's typically served — hot or cold, individually or
family-style, eaten by hand or with utensils — only if relevant to
allergy management (hand-eaten communal dishes carry cross-contact
risk).
- Note the regional variation explicitly if the dish differs
meaningfully across the cuisine ("street-version pad thai has
ground peanut tossed in; restaurant pad thai often serves it on
the side").
- Do NOT advise. Do NOT say "we recommend" or "you should". State
facts the diner can act on.
Output ONLY a single descriptive string. No JSON wrapper, no preamble.
```
---
### Call: TTS speak the waiter phrase aloud in the local language
Model: `gemini-3.1-flash-tts-preview` · n/a · n/a
```
Voice: warm, slightly slow, with the politeness register a guest
would use to a waiter. Pick the Gemini 2.5 Flash TTS voice whose
`languageCode` matches the WaiterPhrase's language_code —
pronunciation will follow that locale automatically.
Pre-process the text before sending it to TTS:
- Read from `phrase_local_language` (the source-language version).
- Insert a single ellipsis (`…`) before the allergen name, so the
TTS produces a small pause that signals importance ("Sou
alérgica … a marisco"). Gemini 2.5 TTS does NOT support SSML
`` — the ellipsis is the textual cue for pace.
- For non-Latin-script phrases, send the native script — the TTS
reads it natively if the languageCode matches. Do NOT send the
Latin transliteration; pronunciation will be wrong.
- Mid-call voice switching is not supported. The phrase is one
language, one voice.
- Target rate: ~120 words per minute — slightly slower than
conversational, so the user can mimic and the waiter can catch
every syllable.
Style direction: prepend ONE short directive sentence to the text
input, exactly like: "Speak warmly, slowly, in the politeness
register a guest would use to a waiter. …". There is no separate
`style` API field on Gemini 2.5 TTS; the directive sentence inside
the input is how style is conveyed.
SSML `` tags are NOT supported on Gemini 2.5 TTS. If a
specific local pronunciation matters (e.g. a regional variant of
the allergen name), trust the voice's native locale. Do not try
to override pronunciation with phoneme tags or with respelling.
The audio is played in the user's earpiece (private) or aloud
(public) — that choice is the client's, not yours.
```
---
### Call: Cross-contact assessment for cuisine + dish + allergen profile
Model: `gemini-3.5-flash` · thinkingLevel: medium · Tools: (none)
```
You receive: the cuisine, the dish, the active allergen profile
(including cross_contact_tolerance), and the kitchen defaults you can
infer from the cuisine.
Your job: assess cross-contact risk and emit an array of
CrossContactRisk records matching the schema.
Hard rules:
- Default to the conservative kitchen pattern for the cuisine. Tokyo
izakaya defaults to one fryer for everything; small Cantonese
kitchens default to one wok; Korean BBQ defaults to a shared
grill; European bistros default to a shared bread basket; many
Indian thalis default to communal serving vessels.
- The user's cross_contact_tolerance modulates severity, not
presence. A single fryer is a single fryer regardless of who is
eating; for an anaphylactic peanut allergy, severity is high; for
an oral-allergy-syndrome user, severity is low.
- Where the cuisine pattern is uncertain (a modern fusion restaurant,
a vegan-only kitchen, a kosher-certified kitchen), say so in
cuisine_specific_note and pick the worse of the plausible
defaults.
- Do NOT speculate about specific restaurants. You do not know this
restaurant. You know the cuisine. Speak in cuisine defaults; the
user asks the kitchen to confirm.
Output ONLY an array of CrossContactRisk objects matching the schema.
No commentary.
```
## 5. Use cases & content to include
Build dedicated UI sections or flows for each of these — they tell you what content the app must support.
- **The Lisbon shellfish case (universal anchor).** Marta, 34, on her second night in Lisbon, allergic to shellfish (anaphylaxis). She lifts her phone over the chalked menu at a tasca in Alfama. Within four seconds *bacalhau à brás* glows green, *amêijoas à Bulhão Pato* glows red, *polvo grelhado* glows orange. Tapping the red dish gives her: "Desculpe — sou alérgica grave a marisco. Tem outro prato sem amêijoas que recomende?" Below the phrase: the literal back-translation, a small audio play button, and a "show this to the waiter" card she taps to flip the phone into a large-print mode.
- **The Tokyo coeliac case.** Aiko and her eight-year-old in a Shibuya ramen counter; the eight-year-old has coeliac disease. The Live overlay reads the four-dish menu in 黒板 chalk. Every ramen is red — the broth is soy-sauce-finished and the noodles are wheat. One side dish (*edamame*) is green. The waiter phrase: "息子は小麦アレルギーがあります。グルテンフリーのメニューはありますか?" with the polite ます-form and the literal back-translation ("my son has a wheat allergy; do you have a gluten-free menu?").
- **The Marrakech peanut case.** Idris, 19, peanut-anaphylactic, on a school trip in Marrakech. The menu is in Arabic script and French. Pastilla is red — almond is named, peanut is plausible in the regional variant. Tagine d'agneau is orange — likely safe, but ground peanut can appear in some berber recipes. The app provides the Moroccan Arabic phrase to ask the kitchen: "عندي حساسية شديدة للفول السوداني — هل هاد الطبق فيه؟" with French as a fallback ("J'ai une allergie sévère à l'arachide — y en a-t-il dans ce plat?").
- **The diaspora Tamil returnee.** A second-generation Tamil-American in Chennai for a family wedding, tree-nut-allergic. She speaks Tamil but does not read modern menu scripts. The app reads the Tamil-script menu, surfaces dish-by-dish ingredients, and produces the Tamil waiter phrase in formal register that her uncle approves of when she shows him.
- **The four-person family table.** A British family in Athens — mother lactose intolerant, father fine, daughter peanut-allergic (anaphylaxis), son fine. Each plate icon next to a dish shows four small dots, coloured per profile. The mother sees two safe options for everyone, three safe for everyone but her, six unsafe for the daughter, the rest safe.
- **The street-food stall.** A traveller at a Bangkok night market with no menu — just a sizzling wok and a chalkboard. The user shows the chalkboard to the camera; the app reads what's there, asks the user to point at the wok, and offers the phrase to ask the cook to clean the wok ("เมื่อกี้ผัดอะไรในกระทะนี้ครับ?" — "what did you just cook in this wok?").
- **The accidental allergen on a buffet.** A wedding buffet in Lagos; the user is dairy-allergic. The Live camera sweeps across labelled dishes (jollof rice, suya, moin moin); the moin moin is orange because the binder can include milk; the app provides the Yoruba phrase and a Pidgin English fallback.
- **The "I asked the kitchen" override.** The user tapped a red dish, asked the waiter, and confirmed it's safe. They tap a small "I asked — confirmed safe" button. The verdict moves to a special outlined green ("safe — confirmed at the table") that never appears for any other dish.
- **The symptom-log incident.** Three weeks ago in Hanoi, the user got a mild reaction at a phở place. They logged it. Tonight in Saigon, they are at a different phở place. The app's first opening message: "You logged a reaction at a phở restaurant in Hanoi on 14 May. Want to add a note about this kitchen before ordering?"
## 6. Page structure
Build the following screens / sections in this order. Adjust copy to fit the voice, but keep the structural intent.
1. **Welcome / sign-in.** A photographed-looking image of a hand holding a phone over a hand-chalked Portuguese menu on a tiled tasca counter at dusk; an empty glass of vinho verde at the edge of frame. One paragraph: "Allergen Menu reads any printed or chalked menu in any language, marks the dishes you can eat, and gives you the local-language phrase to ask the waiter about the ones you can't." Single Google sign-in button; Apple sign-in next to it. Below: "Try with the sample menu" → loads the demo Lisbon menu from section 8a.
2. **Allergen profile setup.** A friendly form for the user's first profile. Picker for allergens; severity slider per allergen; cross-contact tolerance slider; preferred language. A second screen: "Add another diner" for the table case. The default profile is the user's own; named profiles for children include the child's age.
3. **Live menu overlay (mobile-first).** The hero screen. The camera viewfinder fills the screen; overlay rectangles appear on top of every parsed dish, coloured green / orange / red with a small confidence chip. Sticky bottom bar: profile selector ("Marta · Kai (7)"), language indicator ("reading Portuguese"), one-tap "freeze frame" for a tricky menu. Top-right: a small "(i)" with the active profile summary.
4. **Static menu capture flow.** Used when the user prefers a single still photo. Live viewfinder with menu-shaped crop guides; capture; "is the menu more than one page?" prompt; multi-shot bundling; submit. While processing: a calm step-by-step indicator ("Reading the Portuguese…" → "Mapping the dishes to your profile…" → "Generating the waiter phrases…"). 6-15 seconds total.
5. **Menu detail view.** After Live or static capture, the parsed menu in a scrollable layout. Each section is a heading in the source language with a small user-language gloss below. Each dish is a row: dish name verbatim, optional user-language gloss, a coloured chip (green / orange / red), and a tappable "(i)" that opens the per-dish panel. The panel shows: ingredients with confidence chips, cross-contact risks, the waiter phrase if one was generated, the swap suggestion if one was generated, an audio play button for the phrase.
6. **Dish detail panel.** Slides up from the bottom (full-screen on mobile, side-sheet on desktop). At the top: dish name verbatim large; user-language gloss small. Below: the verdict (with a calm explanation), the ingredients table, the cross-contact risks (each as a calm card), the waiter phrase (large, with audio play, with copy-to-clipboard, with "show to the waiter" full-screen flip), the swap suggestion. Below that: the "tell me about this dish" three-sentence description. Bottom: "I asked the kitchen — confirmed safe" toggle.
7. **Show-to-the-waiter card.** A full-screen, high-contrast card with the local-language phrase in large type, the literal back-translation in smaller type below, and an audio play icon. Designed to be readable across a table in dim restaurant light. Single tap dismisses.
8. **Table view (multi-diner).** Magazine-grid of the menu's dishes, each card showing dot-per-diner coloured chips so each person at the table can see at a glance which dishes work for everyone, for some, for nobody.
9. **Travel pack picker.** Before a trip, the user picks a city ("Lisbon"), picks the allergens that matter on this trip (defaults to active profile), and downloads the pack: the canonical local-language allergen vocabulary, common hidden sources, the waiter phrases pre-rendered, the emergency phrase, the local emergency number. The pack is ~5 MB; works offline through the trip.
10. **Trusted restaurants list.** A starred list of restaurants where the kitchen handled the allergy well, with the city, the date, and a one-line note. Tappable; shared across devices if family-mode is on. Symptom-log entries are kept separately and never shared.
11. **Symptom log.** A private, encrypted-at-rest list of reactions: date, city, restaurant, dish, severity, notes. The app uses these to warn the user before reopening a similar restaurant. Never shared.
12. **Settings & profiles.** Profile management; family-mode invite (magic-link email); device-side cache management ("Lisbon pack — 4.7 MB — delete"); allergen-vocabulary update ("update Tokyo pack — new izakaya defaults added by the curator team"); delete-my-data.
13. **Footer.** "Made for the meals that should not put you in the hospital." Privacy: "Your profile is yours. We never train on it." Capabilities `(i)` icon in header.
## 6b. First-visit onboarding
Show a **first-visit onboarding** the first time a visitor lands on the app (detect via `localStorage` flag; do not show on return visits). Three slides, dismissible at any time. Persistent re-entry: a `?` icon in the header reopens it.
**Slide 1 — What this is.**
- Headline: "Welcome to Allergen Menu."
- Subhead: "Eat safely in any city, any language, with every allergen, swap, and waiter phrase ready to read."
- One paragraph (≤ 60 words) explaining who this is for: travellers with serious food allergies, parents of allergic children, coeliacs in cuisines where gluten is a default condiment. What it does differently from a generic menu translator: it never green-lights an ambiguous ingredient, and it gives you the local-language phrase to ask the waiter — not just a translation.
- Visual: a small annotated illustration of a phone over a chalked menu, with three dishes highlighted in green / orange / red and a small "ask the waiter" speech bubble.
**Slide 2 — Try it now.**
- One short prompt: "Try with the sample Lisbon menu".
- A live demo input pre-loaded with the demo menu from section 8a.
- 1-2 sentences pointing at *the specific page elements* where the Gemini magic happens (the Live overlay sweep, the source-language waiter phrase, the cross-contact note on the single-fryer kitchen).
**Slide 3 — How to remix this.**
- Headline: "Make this yours."
- Three short bullets:
- "Swap the sample menu in `/data/seed-menus/` for your own scans."
- "Adjust the cuisine knowledge files in `/data/cuisine-packs/` for cuisines this template doesn't yet cover."
- "Wire up your Gemini API key and Firebase project via the env-var list in the capabilities panel."
- Primary CTA: "Use this template" → links to AI Studio Build remix entry point.
- Secondary: "Just exploring — close" (sets localStorage flag, never auto-shows again).
**Accessibility:** focus trap, `Esc` closes, `role="dialog"`, `aria-modal="true"`, `aria-labelledby`, focus restored to trigger on close. Respect `prefers-reduced-motion`.
**Don't:**
- Don't gate content behind the modal. The page beneath must be fully usable.
- Don't auto-reshow on return visits. Use `localStorage['onboarding-seen-v1']`.
- Don't include unrelated CTAs (newsletter signup, social follow). Keep it about the template only.
## 6c. Capabilities info button (persistent in header)
Add a persistent `(i)` icon in the top-right of the header (next to the primary nav). Click → opens a modal/panel titled **"What powers this app"**.
**Panel contents (in this order):**
**Gemini capabilities used (the hero list):**
- **Gemini Live API (multimodal)** — streams the camera feed and emits per-dish verdicts in near-real time as you sweep across the menu. The hero interaction. First verdict back within ~1.5 seconds.
- **Gemini 3.5 Flash (multimodal, static parse)** — fallback path. One photograph of a menu, one structured response with every dish.
- **Gemini 3.5 Flash (multilingual reading)** — reads any printed or chalked menu in Portuguese, Spanish, French, Italian, German, Greek, Turkish, Arabic, Hebrew, Farsi, Urdu, Hindi, Bengali, Tamil, Punjabi, Sinhala, Khmer, Thai, Vietnamese, Tagalog, Mandarin, Cantonese, Korean, Japanese, Amharic, Swahili — in one call.
- **Gemini 3.5 Flash + grounded search** — answers focused cuisine questions ("does pad thai in Bangkok typically contain peanuts?") with citations.
- **Gemini TTS** — speaks the waiter phrase aloud in the local language, in the politeness register the cuisine expects.
- **Firebase Auth** — Google and Apple sign-in. Magic-link email for family-mode profile sharing.
- **Firestore** — stores your profiles, trusted restaurants, and (encrypted) symptom logs; syncs across your devices in real time.
- **Firebase Storage** — keeps the photographs of menus at upload resolution.
- **Cost note** — see the detailed breakdown in 6d. A typical trip of 14 meals over a week costs about $0.40 of Gemini API spend, total.
- **Privacy note** — your allergen profile, your trusted restaurants, and your symptom logs are private to you and the family members you invite. This app uses the Gemini API on the paid tier, where Google does not use your content for model training, per the Gemini API Additional Terms. Symptom logs are encrypted at rest with a per-user key.
**Backend services this app depends on:**
- Auth: see section 4b. **Apple sign-in requires user configuration** (Apple Developer account + Service ID + Key ID + private key). Magic-link email **requires the sender domain to be authorised in Firebase Auth**.
- Database: see section 4b
- Storage: see section 4b. **Firebase Storage is NOT auto-provisioned by AIS Build today** — enable it in the Firebase console and wire the bucket name in before first capture.
- Email: see section 4b
- Payments: see section 4b (not used in v1)
- External APIs: see section 4b
**Environment variables you'll need to configure:**
- `GEMINI_API_KEY` — your Google AI Studio API key
- `FIREBASE_PROJECT_ID` — your Firebase project id
- `FIREBASE_SERVICE_ACCOUNT` — service-account JSON (server-side only)
- `FIREBASE_STORAGE_BUCKET` — your enabled Firebase Storage bucket
- `SYMPTOM_LOG_ENCRYPTION_KEY` — server-managed; rotate via the Settings screen
**Cost + privacy notes:**
- The Live session is billed per second of streamed input; a typical menu sweep is 8-20 seconds, ~$0.02 per sweep at Gemini 3.5 Flash pricing.
- The static menu parse is billed per token of image input + output, ~$0.015 per menu.
- TTS is billed per output token, ~$0.001 per waiter phrase. Cached after first play, so a re-listen is free.
- Where the data lives: your Firebase project. How to delete it: Settings → "Delete all my data forever" — gone in 60 seconds.
**Documentation links:**
- AI Studio Build docs
- Gemini Live API docs
- Gemini API multimodal, multilingual, grounded search, TTS docs
- Firebase Auth, Firestore, Firebase Storage docs
**Accessibility:** same standards as the onboarding modal — focus trap, `Esc`, ARIA, restored focus.
**Behaviour:**
- Always available — single click from anywhere in the app.
- Tooltip on the `(i)` icon: "How this app is built".
- Mobile: opens as a full-screen sheet that slides up.
- Should be the most honest part of the app — never hand-wave service requirements; never say "AI" without naming the specific Gemini model and capability.
## 6d. Detailed cost breakdown (deployer reads this BEFORE shipping)
- **Live overlay session (Gemini 3.5 Flash Live API)** — billed per second of streamed input. A typical menu sweep takes 8-20 seconds and emits 6-25 DishVerdict events. ~$0.02 per sweep.
- **Static menu parse (Gemini 3.5 Flash, medium thinking)** — typical single-page menu ≈ 1 image input, ~1,200 output tokens (verdicts for 18 dishes). ~$0.015 per menu.
- **Per-dish cuisine knowledge (Gemini 3.5 Flash + grounded search)** — ~$0.002 per dish question. A typical meal triggers 3-5 such questions. Cached per-dish-per-city forever.
- **Waiter phrase generation (Gemini 3.5 Flash, low thinking)** — ~$0.001 per phrase. Cached per-dish-per-language. A 20-dish menu generates maybe 6 phrases (red and orange dishes only).
- **TTS narration of waiter phrase (Gemini 2.5 Flash TTS)** — billed per output token (~$10/M output tokens), effectively ~$0.000003/character. A 15-word Portuguese phrase ≈ $0.001 per render. Cached per-phrase-per-voice; charged once.
- **Dish description "tell me about this dish" (Gemini 3.5 Flash, low thinking)** — ~$0.0005 per dish. Generated lazily, only when the user opens the panel.
- **Cross-contact assessment (Gemini 3.5 Flash, medium thinking)** — ~$0.002 per dish. Generated once per dish-per-cuisine-per-profile.
- **Expected per-meal cost on first use:** ~$0.03 for a Live sweep + 6 waiter phrases + 4 grounded cuisine lookups + 2 TTS renders. **A week of 14 meals:** ~$0.40, dropping after cuisine knowledge caches.
- **Image storage:** Firebase Storage standard tier, ~$0.026/GB/month. A typical menu photo is ~1.5 MB; 50 menus per year ≈ 75 MB ≈ ~$0.002/month.
## 7. Design language
- **Mood:** A trusted travel companion that does one thing very well at a moment when getting it wrong has a body cost. Not a foodie magazine. Not a guidebook. The dimly-lit tasca at 9 pm, the chalked menu the user cannot read, the waiter waiting, the partner already two glasses of vinho verde deep. Calm. Decisive. The colour is the verdict; the words are the conversation.
- **Typography:** A neutral humanist sans for app chrome and dish names (Inter or Geist), pinned at a slightly larger body size than typical (17 px on mobile) because the user is reading in dim restaurant light. A subtle display serif (Source Serif Pro or Newsreader) for the dish-name verbatim line in the detail panel — to honour the menu's voice. A monospace (JetBrains Mono) for the ingredient confidence chips and the IPA pronunciation guide.
- **Palette:** Off-white background `#FAFAF7` for everything; deep ink `#15171C` for body text. Verdict colours are calm, not aggressive: safe green `#2E7D5B` with a soft `#E3F2EA` tint background; ambiguous orange `#B8651F` with a `#FBEDDC` tint; unsafe red `#A8362F` with a `#FBE7E5` tint; confirmed-safe outlined green (border only, no fill). A muted accent slate `#3F4756` for app chrome.
- **Imagery:** The photographs of menus are the hero of capture flows; they are always shown at full bleed when reviewing a parse. Cuisine-card illustrations are line drawings in deep ink — a single fryer, a single wok, a shared bread basket — never photographs of restaurants, never stock food shots. No flame emoji, no marketing food porn.
- **Hand-feel touches:** The Live overlay's verdict chips fade in with a tiny scale-up (0.95 → 1.0, 200 ms) when a verdict first appears; they do NOT animate when a verdict changes from orange → red (that's a hard cut — the user must register the change instantly). The "show to the waiter" card flips with a single-axis rotation transition (300 ms) — like turning a card around at the table. The waiter-phrase audio play button is a soft pulsing dot when audio is playing; a static dot otherwise.
- **Spacing:** consistent 4-px base. Generous around verdict chips so they read as decisive, not crowded.
- **Radius:** consistent token set (e.g. 6 / 12 / 20 px). Verdict chips use 6; the show-to-the-waiter card uses 20; the menu detail rows use 12.
- **Shadows:** subtle, layered, neutral. Avoid coloured shadows; they fight the verdict colours.
- **Motion:** purposeful — verdict fade-ins, panel slide-ups. Respect `prefers-reduced-motion`. No bouncing splash animations. The verdict colour change from orange → red is a hard cut even with reduced-motion off, because the meaning is "instant attention". Reduced-motion converts the panel slide-ups to fade-only.
- **States:** every interactive element has hover, focus, active, disabled. Loading uses skeletons that match the parsed menu layout, not spinners. Empty states have helpful next-action guidance ("Point the camera at the first page of the menu to start").
## 8. Content generation rules
- Write **realistic, specific copy**. NO Lorem Ipsum. NO generic placeholders like 'Your tagline here'.
- Invent plausible dish names, ingredients, waiter phrases, restaurant names that fit the cuisine and city (use the seed content in section 8a as a starting point). When inventing, lean on cuisine-specific patterns — Portuguese tasca defaults, Tokyo izakaya defaults, Marrakech tagine defaults, Hanoi phở-stall defaults — but never claim a fictional waiter phrase as something a specific real waiter would say.
- Tone: warm, direct, calm under pressure. This template is for a person who is sometimes scared at the moment of using it. No corporate language. No hype.
- Headlines: punchy and concrete. No 'Empower your X' filler. No 'Revolutionize'. No 'Seamless'.
- Body copy: short paragraphs (2-4 sentences). Use lists where appropriate.
- Plain language. Avoid jargon — except where the user already speaks the jargon (the coeliac user wants to see "gluten" and "wheat" precisely; the cross-reactivity-aware user wants to see "lipid transfer protein").
- Where the app outputs AI-generated content, never label it as "AI says" — let it speak. Use small uncertainty cues only where epistemic honesty requires them (a low-confidence ingredient chip shows a small confidence number; tapping it reveals "we couldn't fully resolve *amêijoas à Bulhão Pato* — the verdict is conservative").
## 8a. Seed content (use these specific examples)
Anchor every generated copy + sample data point in the concrete content below. Use these names, numbers, and snippets verbatim where helpful, or generate close variants that sit in the same world.
**Sample profiles (sidebar):**
- "Marta — shellfish (anaphylaxis), zero cross-contact tolerance, English."
- "Kai (7) — coeliac (severe), low cross-contact tolerance, English."
- "Aiko — peanut + tree nut (anaphylaxis), zero cross-contact tolerance, Japanese."
- "Idris — peanut (anaphylaxis), zero cross-contact tolerance, English + French."
- "Rohini — tree nut (severe) + oral allergy syndrome (birch group), moderate cross-contact, Tamil + English."
**Sample travel packs (loaded into the picker):**
- "Lisbon — shellfish, fish, gluten."
- "Tokyo — gluten, peanut, soy."
- "Marrakech — peanut, tree nut, sesame, dairy."
- "Hanoi — peanut, shellfish, fish, soy."
- "Mumbai — peanut, tree nut, dairy, gluten."
- "Mexico City — peanut, dairy, egg, gluten."
**Sample Lisbon menu in detail view (this is what the demo should show):**
Restaurant name (verbatim): "Tasca da Esquina"
City inferred: Lisbon
Cuisine inferred: Portuguese coastal
Source language: pt-PT
Section: "Entradas" (Starters)
- **Pão com manteiga de azeitona** — "bread with olive butter". Ingredients: bread (wheat), olive butter (dairy). Verdict for Marta (shellfish): green. Verdict for Kai (coeliac): red — wheat in the bread. Waiter phrase for Kai: "Tem pão sem glúten para acompanhar?" (literal: "Do you have gluten-free bread to accompany?")
- **Amêijoas à Bulhão Pato** — "clams with garlic, white wine, coriander". Ingredients: clams (*amêijoas* — shellfish_mollusc), garlic, white wine (alcohol), coriander, olive oil. Verdict for Marta: red. Verdict for Kai: green. Waiter phrase for Marta (swap to *bacalhau à brás*): "Desculpe — sou alérgica grave a marisco. Pode recomendar outro prato sem amêijoas? O bacalhau à brás é seguro para mim?" (literal: "Sorry — I am severely allergic to shellfish. Can you recommend another dish without clams? Is the bacalhau à brás safe for me?")
- **Salada de polvo** — "octopus salad". Ingredients: octopus (*polvo* — shellfish_mollusc), red pepper, onion, olive oil, vinegar (sulphite). Verdict for Marta: red. Verdict for Kai: green.
Section: "Pratos Principais" (Main courses)
- **Bacalhau à brás** — "shredded salt cod with onions, potatoes, scrambled egg". Ingredients: salt cod (fish), onion, potato (matchstick fried), egg. Verdict for Marta (no fish in profile): green. Verdict for Kai (coeliac): orange — the matchstick potatoes are typically fried in a shared fryer; ask the kitchen. Cross-contact note: "single-fryer kitchen — confirm fryer history with the kitchen". Waiter phrase for Kai: "As batatas são fritas no mesmo óleo que pratos com glúten? O meu filho é celíaco." (literal: "Are the potatoes fried in the same oil as gluten dishes? My son is coeliac.")
- **Polvo à lagareiro** — "octopus with smashed potatoes, olive oil, garlic". Ingredients: octopus (shellfish_mollusc), potato, olive oil, garlic. Verdict for Marta: red. Verdict for Kai: orange (fryer cross-contact on the potato).
- **Bife à café** — "steak in a creamy mustard sauce". Ingredients: beef, butter (dairy), mustard (mustard), garlic. Verdict for Marta: green. Verdict for Kai (coeliac): orange — mustard sauces sometimes use wheat flour as binder; ask the kitchen.
Section: "Sobremesas" (Desserts)
- **Pastel de nata** — "egg custard tart". Ingredients: puff pastry (wheat), egg, milk (dairy), sugar, cinnamon. Verdict for Marta: green. Verdict for Kai: red — wheat in the pastry. Waiter phrase: not generated (no plausible swap — the pastry is structural).
**Global cross-contact notes for this menu:**
- "Kitchen uses one fryer for all fried items. For coeliac (Kai), every fried item is orange at minimum."
- "Bread basket is shared across tables. For coeliac, the basket is removed before seating on request."
**Emergency phrase (in pt-PT):** "Estou a ter uma reação alérgica grave. Por favor chamem uma ambulância — 112." (literal: "I am having a severe allergic reaction. Please call an ambulance — 112.")
**Local emergency number (Portugal):** "112"
**Sample Tokyo menu (second seed):**
Restaurant name (verbatim): "Shibuya Yokocho Ramen"
City inferred: Tokyo
Cuisine inferred: izakaya / ramen counter
Source language: ja-JP
Section: "麺類" (Noodles)
- **醤油ラーメン** (shōyu ramen) — "soy-sauce ramen". Ingredients: wheat noodles, soy sauce (contains wheat in most commercial brands), pork broth, chashu pork, spring onion, nori. Verdict for Kai (coeliac): red — both noodles and soy sauce contain wheat. Verdict for Aiko (peanut + tree nut): green. Waiter phrase for Kai: "息子は小麦アレルギーがあります。グルテンフリーのメニューはありますか?" (literal: "My son has a wheat allergy. Do you have a gluten-free menu?") — most Tokyo ramen counters will say no; the phrase exists so the user can ask without giving offence.
**Sample Marrakech menu (third seed):**
Restaurant name (verbatim): "Café Argana, Jemaa el-Fna"
City inferred: Marrakech
Cuisine inferred: Moroccan
Source language: ar-MA + fr-FR (bilingual menu)
Section: "Tajines"
- **Tajine d'agneau aux pruneaux** — "lamb tagine with prunes and almonds". Ingredients: lamb, prunes, almonds (tree_nut), onion, ginger, cinnamon. Verdict for Idris (peanut, not tree nut): green — confirmed no peanut, almonds are tree nut not peanut. Verdict for Rohini (tree nut + birch OAS): red — almonds named.
**Sample voice copy:**
- Onboarding: "Read any menu in any language. We mark the dishes you can eat and give you the words to ask the waiter about the ones you can't."
- Processing: "Reading the Portuguese…" / "Mapping the dishes to your profile…" / "Writing the waiter phrases in Portuguese…"
- Empty menu: "Point the camera at the first page of the menu to start. We'll mark each dish as you sweep across."
- Error (couldn't read): "We couldn't read this menu clearly — try a closer photo, or tap any dish you can read to ask about it directly."
- Save confirmation: "Saved this menu — *Tasca da Esquina*, Lisbon, May 23."
- Sticky-red warning: "We marked this dish red. Ambiguity is not safe — ask the waiter before ordering."
- Confirmed-safe: "You confirmed with the waiter. We'll show this dish as confirmed-safe for this meal only."
**Sample family-mode invitation email:**
- Subject: "Marta — I added Kai's coeliac profile to our trip. Want to add yours?"
- Body: "I added Kai's allergens to our Allergen Menu profiles before Lisbon. Tap to join the family-mode share — your own profile stays private to you; we only share each other's allergen profiles so the menu colours work across our table." [Open Family Mode]
## 9. Media & assets
- **Hero image (landing screen):** A photographed-looking shot of a hand holding a phone over a hand-chalked Portuguese menu on a tiled tasca counter at dusk; an empty glass of vinho verde and a small dish of olives at the edge of frame. Generate via Nano Banana 2 with a prompt emphasising "tiled counter, warm overhead light, chalked menu, a phone held over it casting a small shadow, no people's faces, late evening warmth, slight imperfection in focus".
- **App icon / wordmark:** Set in the humanist sans. A small green dot and a small red dot beside the wordmark.
- **Empty-state illustration:** A simple line drawing of a phone hovering over a menu, with three dishes highlighted in green / orange / red. Hand-drawn aesthetic, not a flat icon.
- **Demo menu photographs:** Generated per the prompts in section 8a — Nano Banana 2 prompts that specifically request "chalked Portuguese menu on slate, soft restaurant light, slightly crooked, no people in frame", "izakaya counter menu in vertical hand-written kanji on weathered wood", "bilingual Arabic-French printed menu on a brass tray, Marrakech café evening light, no people in frame". Each demo menu should look photographed, not rendered.
- **Cuisine-card illustrations:** Line drawings of the single-fryer izakaya, the single-wok small Cantonese kitchen, the shared bread basket bistro, the shared grill at a Korean BBQ table. One line, deep ink, no shading.
- **Stock fallbacks:** If image generation fails, fall back to the photographed sample menu from `/public/samples/sample-menu-lisbon.jpg`. Never to a "🍽️" emoji.
- **Generated imagery:** prefer Nano Banana 2 over stock photography. Prompt for warmth, asymmetry, and slight imperfection — avoid the glossy 'AI render' look.
- **Optimisation:** WebP/AVIF, `loading="lazy"`, explicit `width`/`height` to prevent layout shift.
- **Icons:** `lucide-react` for UI. Use sparingly — never decorative-only.
### Build-time asset manifest (explicit specs)
Every image, illustration, and visual reference mentioned above must resolve to ONE of the three buckets below — runtime-generated, seed-shipped, or user-supplied. Do NOT ship `` tags whose `src` is not listed here. Do NOT depend on bare "section 8a prompts" without binding them to explicit paths and model IDs.
**Bucket 1 — Runtime-generated (Nano Banana Pro `gemini-3-pro-image` for hero/demo photographs; Nano Banana 2 `gemini-3.1-flash-image` for in-app illustrations and reference-conditioned variants).** Cached to Firebase Storage; served via signed URL. Every reference above to "Nano Banana 2" or "Nano Banana Pro" MUST be wired to one of these specific calls with an explicit model id:
- `/public/generated/hero.webp` (2400×1500, WebP) — model `gemini-3-pro-image` — uses the literal prompt described as "Hero image (landing screen)" above. Run once at build; commit a `/public/samples/hero-fallback.webp` (1600×1000) generated from the same prompt with `gemini-3.1-flash-image` so the page renders if quota is exhausted.
- `/public/generated/demo/{demo-slug}-{NN}.webp` (1600×1200, WebP) — model `gemini-3.1-flash-image` (reference-conditioned where the prior frame is passed as input) — one path per "Demo X" image referenced above. The slug derives from the seed example in section 8a; the NN index covers each frame in the demo sequence.
- `/public/generated/illustrations/{name}.webp` (1024×1024, WebP) — model `gemini-3.1-flash-image` — one path per named illustration above ("Empty-state illustration", "Recipe-card hero illustrations", "Curriculum picker imagery", "Period-style frames", etc.). Each illustration's prompt is the literal description above; ship a deterministic seed in the request so re-runs are reproducible.
**Bucket 2 — Seed assets shipped with the deliverable.** Every "Stock fallback" path referenced above (e.g. `/public/samples/sample-X.jpg`) is generated once via Nano Banana 2 (`gemini-3.1-flash-image`) at 1024×1024 WebP using the same prompt as its Bucket-1 counterpart, then committed to the repo so the page renders identically if Gemini quota is exhausted or the user is offline. Replace any `.jpg` extension above with `.webp` to match the optimisation rule. Also commit these empty-state seeds (1024×1024 WebP, single-stroke hand-drawn line, no colour fill):
- `/public/samples/empty-state-primary.webp` — line drawing of the app's primary empty surface (the named "Empty-state illustration" above), generated from that exact prompt.
- `/public/samples/empty-state-archive.webp` — line drawing of an empty saved/archive view, single-stroke outline.
- `/public/samples/empty-state-error.webp` — line drawing of a hand placing a single object aside with care, used when an AI call fails.
**Bucket 3 — User-supplied.** Uploads from the user's camera / file picker land at the Firebase Storage path conventional for this template (named in section 4b). The build ships with Bucket-1 + Bucket-2 only; no user-supplied images at first paint.
**Hard rules**
- Every `` tag MUST have a `src` that resolves to a path listed in Bucket 1, Bucket 2, or a Bucket 3 upload path. Anything else is a build error.
- No bare `image.jpg` / `hero.jpg` / `placeholder.png` references anywhere in the code.
- Model IDs: `gemini-3-pro-image` for hero-quality photographic generation; `gemini-3.1-flash-image` for in-app illustrations, reference-conditioned variants, empty-state seeds, and stock fallbacks. Never use a legacy model id (no `imagen-*`, no `gemini-1.5-*-image`).
- File format: WebP everywhere (AVIF acceptable where the target browsers support it). No `.jpg` / `.jpeg` / `.png` in `/public/samples/`.
## 10. Interactivity & states
- Every interactive element has hover, focus, active, and disabled states.
- Forms validate inline and show specific error messages (not "Invalid input").
- Loading states use skeletons that match the eventual layout, not spinners.
- Empty states explain the next action with a button whose label fits THIS app's domain: "Point the camera at the menu", "Add a profile for the kids", "Download the Lisbon pack" — never a generic "Add your first item".
- Smooth scroll for in-page anchors.
- Live overlay verdicts fade in (200 ms) when they first appear; verdict changes from orange → red are hard cuts (no animation) because the meaning is "instant attention".
- The Live overlay is debounced server-side: a verdict updates only after two consistent events 400 ms apart, and a red verdict is sticky (red → green requires three consistent green events plus a user confirmation tap).
- If a Live session call fails, drop to the static photo capture path with a calm message ("The live overlay paused — point the phone at the menu and tap to capture a still") and offer retry.
- If a static parse fails, show a specific error ("We couldn't make out the chalked text on this menu — try a closer photo, or tap a dish you can read to ask about it directly") and offer retry with capture guides.
- Low-confidence ingredient chips are faintly hatched; tapping reveals the model's confidence and the alternates it considered.
- The "show to the waiter" card flips with a single-axis rotation transition (300 ms) and locks the phone into landscape with the screen brightness raised so it's legible at the table; `prefers-reduced-motion` swaps the flip for a fade.
## 11. Tech & responsive requirements
- **TTS markdown-stripping preprocessor:** before sending any user-authored markdown to `gemini-3.1-flash-tts-preview`, strip non-spoken markdown: `#`/`##`/`###` headings (keep the title text), `**bold**` (keep the inner text), `[label](url)` (keep `label`, drop URL), `` ``` `` fenced code blocks (skip entirely), `>` block-quote markers (keep the text), and `|` table pipes (read row-by-row as sentences). Insert `…` between sentences for a short pause and a blank line plus `—` between paragraphs for a long pause. The model does not understand markdown; raw markdown will be read aloud as literal characters ("asterisk asterisk").
- **File downloads on Safari / Firefox:** when offering local-disk save of any export (PDF, CSV, MP3, ZIP, JSON, image), fall back to `` with a blob URL — the File System Access API (`showSaveFilePicker()`) is Chromium-only. Detect with `'showSaveFilePicker' in window`; otherwise use the anchor-download path.
- **Stack:** React + TypeScript + Tailwind CSS. Functional components + hooks. Use Shadcn UI primitives where appropriate.
- **Build runtime:** AI Studio Build — full-stack with Cloud Run server-side functions. All Gemini API calls happen server-side; API key lives in Secrets Manager, never in client bundle. The Live API session is proxied through a Cloud Run streaming endpoint; the client sends camera frames over WebSocket, the server holds the Gemini session and forwards the structured verdicts back.
- **Model selection:** explicitly pin `gemini-3.5-flash` for the Live overlay session, the static menu parse, the waiter-phrase generation, and the cross-contact assessment. Pin `gemini-3.5-flash` for grounded cuisine questions and dish descriptions. Pin `gemini-3.1-flash-tts-preview` for the waiter-phrase audio. Set `thinkingLevel` explicitly per call (omit on TTS).
- **Database:** Firestore (auto-provisioned by AI Studio Build). Show the seed Lisbon menu and three sample profiles on first launch.
- **Auth:** Firebase Auth — Google sign-in by default; Apple sign-in next to it; magic-link email for family-mode share.
- **Storage:** Firebase Storage for menu photographs. Pre-signed URLs only. Enable the bucket in the Firebase console before first capture; AIS Build does not auto-provision Storage.
- **Mobile-first.** Verify layouts at 375 px (iPhone SE), 768 px (iPad), 1024 px, 1440 px+. The Live overlay viewfinder must work cleanly in portrait at 375 px because that is the canonical use case.
- Use `clamp()` for fluid typography. Prefer container queries over media queries for component-level responsiveness.
- Use `dvh` / `svh` instead of `vh`. Respect safe-area insets on iOS — the bottom profile selector must sit above the home indicator.
- Zero horizontal overflow at any width. Zero layout shift on load.
- Persist user data in Firestore. Use real-time listeners on the active-profile view so family-mode updates appear instantly.
- Optimistic UI on writes; reconcile on response.
- Long-context guardrail: keep the Live session scoped to the active menu and the active profile only. A user with 200 historical menus does not need them in the session context — load history on demand. For the static-parse path, a single menu averages ~2,000 tokens; well under the 1M ceiling.
- Camera access: the Web Camera API with explicit user permission; permission denied falls back to the photo-library upload path.
- **iOS Safari gotchas (graceful degradation):** the Live API session must handle iOS audio-session interruption — pause on `MediaStreamTrack.onmute`, surface a calm "the call paused for a moment — resume" affordance, rebuild on `onunmute`; backgrounded Safari tabs throttle the WebSocket and kill `getUserMedia` — pair `visibilitychange` with a screen Wake Lock during a meal so the verdict stream survives a dim-screen moment; camera and mic permissions do NOT persist across page reloads on iOS — re-request on every meal; on Low Power Mode iOS may degrade camera resolution — always offer the photo-library fallback so a verdict still works; rotation drops the camera track on iOS — re-bind on `orientationchange`.
- WebSocket reconnect: if the Live session drops mid-meal, reconnect with the same session state and resume from the last verdict.
## 12. Accessibility (WCAG 2.2 AA)
- Semantic HTML — `header`, `nav`, `main`, `section`, `article`, `footer`.
- All interactive controls reachable by keyboard with a visible focus ring.
- Color contrast ≥ 4.5:1 for body, 3:1 for large text and UI components. Verdict colours pass against the off-white background; the red is calibrated to be distinguishable for the most common red-green colour blindness types, and the verdict ALWAYS carries a text label and an icon in addition to the colour (a checkmark, a question mark, a cross) — never colour alone.
- All images have meaningful `alt` text. The menu photographs have `alt` describing the artefact ("photograph of a chalked Portuguese menu at a Lisbon tasca, four sections visible, evening light").
- Form fields have associated `