AR Operator AssistAugmented Machine Guidance
UI / UX Concept · v1

Point. Ask. Be guided through the machine — step by step.

The operator raises a question by voice or text and points the phone at the console. The app recognizes the screen and overlays exactly which control to touch — correcting them in real time if they stray off-path.

Download the Android demoAPK · Android 7+ · ~50 MB
PROJECT  AR Machine Assistant
PLATFORM  iOS · Android
RUNTIME   On-prem · No cloud
DATE      2026·06·18
9:415G  ▮▮▮  87%
On-prem · Secure
// Operator · Line 3
What do you
want to do?
⚡ Change temperature Start a new batch Calibrate sensor Safe shutdown
Injection Molder X200● Recognized · console online
Ask or speak a task
01 · Intake

Ask by voice or text

Chatbot-style intake. The app already knows which machine it's looking at, and an on-prem badge keeps the trust signal front and centre.

9:415G  ▮▮▮  86%
Heater · Zone 1 setpoint
180→ 200°C
MODE: MANUAL● READY
MENU
TEMP ▲
Raise heat
TEMP ▼
START
STOP
ESC
CONSOLE · MOLDER X200MATCH 98%
On track
Step 2 of 4
Press TEMP ▲ to raise the Zone 1 setpoint toward 200 °C.
This is the heater increase key — hold it until the display reads 200°C.
I pressed it →
02 · Guidance

The control, highlighted live

A pulsing HUD reticle locks onto the exact key. The instruction card states the move and the reason, with one-tap voice replay. Re-checks after every action.

9:415G  ▮▮▮  86%
⚠ Maintenance · diagnostics
SVCmenu
LOCKED● NOT YOUR STEP
◀ BACK
Go back here
DIAG
LOGS
SVC
CAL
EXIT
CONSOLE · MAINTENANCEOFF PATH
Wrong screen
Off track · paused
This is the Maintenance menu. Press ◀ BACK to return to the main screen.
You've stepped off the temperature task — guidance is paused until you're back on the right screen.
Take me back
03 · Self-correct

Catches a wrong turn

If the operator opens the wrong menu, the app detects it, pauses the task, and guides them back — the differentiator that makes this more than a chatbot.

Discussions

Project conversations and the requirements they produced.

Participants: Yogesh (technical lead — vendor / Archer Infotech) and the client team (Mission, Amar + Sanjay), who have a downstream client and a German partner, Eckstein Business solution.

The concept

An augmented-reality assistive app — “more than a chatbot” — that guides an operator through running an industrial machine. The operator points their phone camera at the machine console, asks a question (typed or spoken), and the app overlays step-by-step guidance on the live view, highlighting the exact knob/button to press and explaining it, until the task is done. (Example: changing a machine’s temperature setting.)

Three system components

  • Mobile app — Flutter (iOS + Android); a passive client that captures camera frames + the query and renders the AR overlays. No business logic on the device.
  • Backend server — the active “brain”: all logic, AI, navigation graphs and decisions.
  • Admin / authoring console — to capture machine console screens, annotate them, and upload to the server to build the knowledge base.

Core flow

Ask → point camera → capture frame → backend recognizes the console and current state → computes the navigation path → app overlays guidance → user acts → backend re-checks they’re on the right path (corrects if off-track) → loop until the goal is reached.

Validation rules

Must be pointed at a known console; console must be powered ON and fully in frame; progress is blocked if not pointed at the machine.

Hard constraints

  • Fully on-premises, on the intranet (IP / confidentiality).
  • No cloud AI — a small-footprint model on the on-prem server.
  • All console screens, navigation graphs and annotations are built from scratch — hence the in-house authoring tool.

Scope

Start with a POC, but even the POC must be end-to-end (mobile + backend + admin + custom AI). Future scope: operator training, fault / anomaly detection, and validation / monitoring (e.g. flagging incorrect readings).

Action items

  • Vendor: a demo app by the weekend — using MS Word (and Gmail) as a stand-in “machine” to prove the concept.
  • Vendor: 1–2 UI/UX mockups (home + one flow screen).
  • Vendor: share team size, UI/UX capacity and portfolio (client flagged UI/UX as top priority).
  • Client: share the formal requirement (Excel) + the backend tech-stack sheet.
  • Both: follow-up call; client also expecting a demo / slides from Eckstein Business solution.

Meeting outcome

Both sides aligned to build a from-scratch, fully on-prem POC. Agreed next steps: the vendor delivers a working proof-of-concept demo (MS Word / Gmail standing in for a machine console) plus 1–2 UI/UX mockups by the weekend; the client shares the formal requirement (Excel) and the backend tech-stack sheet. A follow-up call will review the demo together with slides from the German partner, Eckstein Business solution.

Flutter client 100% on-prem · no cloud AI

Upload reference images

Add a screenshot for a screen so the app recognizes it on camera. Saved on the server; recognition reloads immediately — no app rebuild.

App mode
Click to choose or drop an image here

Images are stored on the server’s persistent volume and recognition reloads automatically. The admin key is required — uploads are not public.

Uploaded references