Amy AI — local-first desktop assistantYour computer, talked to.
Amy AI is a voice assistant that runs on your own machine. She hears you across the room, looks at whatever is on your screen, drives your apps for you, and models real CAD parts — with the language models running locally through Ollama, so nothing is shipped off to an API.
What a local AI assistant can actually do
Amy has hands, eyes and ears on the machine she runs on. Every subsystem below is part of the same process — one wake word reaches all of them.
Full-duplex voice
She listens while she speaks. Interrupt her mid-sentence and she stops — barge-in is voice-activated, with a clap or a global hotkey as backup wake signals.
Read moreAmy listens and speaks at the same time. There is no press-to-talk and no waiting for her to finish — start speaking and she stops mid-sentence.
How to use it
- Say the wake word, then your request: “Amy, what’s the weather?”
- Interrupt her at any point by simply talking over her.
- Clap twice to wake her if your voice won’t carry.
- Or press a global hotkey — works in a noisy room, with music on, or when you simply don’t want to say her name. Press it again while she’s talking to stop her.
- Pause mid-thought and she keeps listening rather than cutting you off.
Settings
wake_word | What she answers to. Pick something that won’t appear in normal conversation. |
barge_in | voice stops her the moment you speak; off lets her finish. |
barge_sensitivity | Raise it if background noise keeps cutting her off. |
clap_wake | Clap detection as a backup wake signal. |
push_to_talk | A system-wide hotkey such as ctrl+alt+a. Empty disables it. |
whisper_model | small.en by default. Larger is more accurate and slower. |
continuation_listening | Keeps the mic open if you paused rather than finished. |
Worth knowing
Install faster-whisper and she uses it automatically for offline recognition — noticeably more accurate than the default. Voice activity detection gates the expensive stages, so the transcriber only runs on real speech.
Screen awareness
A vision model reads your actual display on an interval. Ask what's on screen, what an error means, or what that dialog is asking for.
Read moreA local vision model looks at your actual display, so you can ask about what you are looking at instead of describing it.
How to use it
- “Amy, what am I looking at?”
- “What does this error mean?”
- “What is this dialog asking me?”
- Double-click the desktop orb to make her look at the screen immediately.
Settings
vision_interval | Seconds between automatic looks. Lower is more responsive and more expensive. |
screen_awareness_on_start | Off by default — she only looks when asked. |
lens_tokens | Reply length for vision answers. Short answers are much faster. |
warm_vision | Preloads the vision model at startup so the first look is not slow. |
Worth knowing
This needs a vision-capable model pulled in Ollama — llava by default. Screen captures are processed locally and never leave the machine.
Computer control
An agent that finds and drives real applications through UI automation — opening, clicking, typing. Risky steps wait for your approval first.
Read moreAmy builds an index of the applications on your machine, then finds and drives them through Windows UI automation — opening, clicking, typing, navigating.
How to use it
- “Open Chrome and search for CadQuery docs.”
- “When I say my editor, I mean VS Code” — teaches an alias she remembers.
- “How many apps can you see?” — rebuilds the app index.
Settings
automation_approval | risky gates destructive steps only; all asks before everything. |
approval_timeout | Seconds to wait for your answer. Timing out declines. |
allow_shell_commands | Off by default, deliberately. Only enable if you want her running shell commands. |
Worth knowing
Approval prompts appear in the window and are spoken. Leaving automation_approval at risky is the sensible default — she acts freely on safe steps and stops for anything that deletes, sends or overwrites.
Real CAD, not meshes
CadQuery builds exact B-rep solids — hinged assemblies, tolerances, STEP export — then a visual check confirms the part came out as described.
Read moreMost AI CAD produces meshes you cannot edit. Amy drives CadQuery to build exact B-rep solids — real geometry with real constraints, exportable to STEP for any CAD package.
How to use it
- “Model me a hinged enclosure, 80 by 50 by 30 millimetres.”
- “Make the wall thickness 3 mm and add mounting tabs.” — refines the current part.
- Edit the generated CadQuery code directly in the CAD panel and re-run it.
- Export to STEP or STL from the same panel.
Settings
cad.max_attempts | How many times she retries when the geometry fails to build. |
cad.visual_check | Renders the result and checks it against your description before showing it. |
cad.slicer_path | Point this at your slicer to send parts straight to print. |
Worth knowing
CadQuery is about 450 MB and is imported lazily — it costs nothing until the first CAD request. Parts land in amy_data/cad/.
Cameras & gestures
Run several named cameras — desk, bench, printer — and switch by voice. Point one at something and ask what it is. Gestures map to actions: open palm pauses media, fist mutes.
Read moreRun as many cameras as you have, give each one a name, and switch between them by voice. Point one at something and ask what it is.
Setting cameras up
- “Amy, scan for cameras” — probes your devices and adds any new ones.
- “What cameras do I have?”
- “Rename the desk camera to bench”
- “Switch to the bench camera”
- “Add a camera called printer on device 2”
Or use the dropdown in the camera view to switch, and the pencil to rename.
Using the feed
- “What is this?” — identify an object held up to the lens.
- Read text, translate, explain or solve what is in frame, from the camera bar.
Gestures
| Open palm | Pause media |
| Fist | Mute |
| Point up | Volume up |
| Peace | Next track |
| Thumbs up | Screenshot |
| Shaka | Identify object |
Gestures need the camera running, and are remapped under gestures in the config.
Proactive engine
She thinks on a timer, watches folders for new files, and raises things worth raising — rate-limited, and silent during quiet hours.
Read moreAmy thinks on a timer rather than only when spoken to. She watches folders, notices things worth raising, and tells you — within limits you set.
How to use it
- Suggestions appear as cards you can accept or dismiss.
- Drop a file into a watched folder and she will offer to do something sensible with it.
- “Be less chatty” adjusts how often she speaks up.
Settings
proactive.level | off, low, normal or high. |
quiet_hours | Start and end hour during which she stays silent. Defaults to 23–7. |
max_per_hour | Hard ceiling on unprompted interruptions. |
think_minutes | How often she reflects on what is going on. |
watch_folders | Folders to notice new files in — Downloads and Videos by default. |
Memory that persists
Transcripts, a knowledge base and learned skills live on disk in amy_data, watched for changes and recalled in later conversations.
Conversations, facts and learned skills survive restarts. Everything lives as plain files in amy_data/, so you can read and edit them yourself.
How to use it
- “Remember that my printer bed is 220 by 220.”
- “What do you know about my workshop?”
- “Consolidate your memory” — merges duplicates and prunes stale facts.
- Drop notes or documents into the knowledge base folder and she picks them up.
What is on disk
SOUL.md | Who she is and how she behaves. |
MEMORY.md | The index of what she knows about you. |
HEARTBEAT.md | Running notes between sessions. |
skills/ | Routines she has learned. |
amy_memory.json | Structured facts, with a backup alongside it. |
Worth knowing
amy_data/ is personal and is excluded by .gitignore — do not commit it, and check the debug log for credentials before sharing it.
Routines
Things that run on a schedule, in her own words. “Every day at 8, brief me on the weather” — and a routine can do anything you could say out loud.
Read moreA routine is just a command with a time attached. Because it runs through the same dispatch path as speech, anything you can ask for can be scheduled.
How to set one up
- “Every day at 8, brief me on the weather and my calendar”
- “Weekdays at 9am, check my email”
- “Every evening at 6, summarise what I downloaded today”
- “List routines” — see everything scheduled
- “Remove the routine morning brief”
Worth knowing
Routines run whatever your proactive.level is, including off — a routine is an explicit instruction, not a nudge, so quiet hours and the interruption ceiling don’t suppress it. They live under routines in the config, so you can edit them by hand:
"routines": [
{ "name": "morning brief", "at": "08:00",
"days": "weekdays", "command": "brief me on the weather",
"enabled": true }
]days accepts daily, weekdays, weekends, or a list of day numbers where Monday is 0.
An audit trail
Every plan approved or refused, every message sent, every app driven — written to disk as it happens. Software that can click buttons for you should be able to show its receipts.
Read moreAmy can drive your applications, send mail and write files. That only works if you can check what she actually did, so every outward action is appended to a log the moment it happens.
How to read it
- “What have you done today?”
- “Show me the action log for the last 6 hours”
What gets recorded
approval | Plans you approved, refused, or that timed out unanswered. |
automation | Agent tasks and application control. |
message | Email and texts, with the recipient and subject. |
system | Routines firing, cameras changing, the hotkey being pressed. |
Worth knowing
It lives at amy_data/actions.jsonl — one JSON object per line, flushed immediately, so a crash can’t lose the record of what happened before it. It never leaves your machine, and it is excluded from version control along with the rest of amy_data/.
Also on Android
A second build that runs on the phone itself, so it still works with your PC off. Always listening, drives your apps, and a floating orb over everything else.
Read moreNot a remote control and not a port — a sibling that runs the model, the speech and the memory on the handset.
What it does
- Listens for “Amy” continuously, with a conversation window so follow-ups don’t need her name.
- Runs a small local model with GPU acceleration — and with Gemma 3n it can see, describing photos you take or pictures you attach.
- Opens apps, taps by name and types, through Android’s accessibility service. Root is optional and only adds to it.
- Floats an orb over other apps — drag it, tap to open, long-press to sleep.
- Looks things up on the web when asked, or when a question needs today’s answer, and names the sites.
- Camera capture, file attachments and conversation history.
What does not carry over
CAD, reading your PC’s screen and Windows automation cannot cross to a phone. A handset also runs a 1–4B model against your desktop’s much larger ones, so expect it to be faster but less capable. That is physics, not effort.
Installing it
There is no Play Store build. Open the android/ folder in Android Studio, run it on your phone, then push a model across — Gemma 3n if you want her to see. Full steps are in the Android section.
Model routing
Separate local models for fast replies, reasoning, code, CAD and vision. Auto-select picks the right one per request instead of one model for everything.
Read moreOne model for everything is a compromise. Amy keeps separate models for different jobs and picks per request, so quick questions stay quick and hard ones get a model that can handle them.
The roles
fast | Short replies, acknowledgements, simple lookups. |
chat | Ordinary conversation. |
reasoning | Multi-step problems worth the extra time. |
code | Writing and fixing code. |
cad | Generating CadQuery geometry. |
vision | Screen and camera understanding. |
How to use it
Leave any role blank and it falls back to the main model. Fill in the ones you care about:
"models": {
"fast": "llama3.2:1b",
"reasoning": "qwen2.5:14b",
"code": "qwen2.5-coder:7b",
"vision": "llava"
}Set auto_select to false to pin everything to one model. auto_start launches the Ollama server on boot, and stop_on_exit shuts it down when she closes.
Everything stays home
Inference runs against localhost. Mail, messaging and smart-home integrations are opt-in, and credentials live only in your own config file.
Read moreThe assistant is local by construction, not by promise. Inference goes to localhost, and anything that reaches the internet is something you configured yourself.
What never leaves
- Your voice, and every transcript of it.
- Screen captures and camera frames.
- Conversations, memory and knowledge base.
- Generated code and CAD.
What can reach out, if you set it up
- Email through your own SMTP account.
- SMS through your own Twilio credentials.
- Spotify, Home Assistant, weather and web research.
Where credentials live
Only in amy_config.json, on your machine. The copy in the repository contains placeholders only. If you fork Amy and add real keys, keep that file out of version control.
How an offline voice assistant works, wake word to reply
What happens between you saying “Amy” and hearing an answer back.
- Wake wordMicrophone, clap detection and voice activity detection decide when she is being spoken to.
- Speech to textWhisper or Vosk transcribes locally, on real speech only.
- Intent routerGrammar matching and reference resolution work out what you meant.
- Local model, skills, agent or visionThe request fans out to whichever subsystem can answer it.
- ComposeThe answer is deliberated over and self-checked before it is given.
- Voice and cardA short spoken reply through edge-tts, with the full answer on screen.
How to set up a local voice assistant on Windows
Windows, Python 3.10 or newer, and a machine with enough headroom to host a local model.
Clone and install
CadQuery and OpenCV are heavy, but they're imported lazily — nothing loads until you ask for CAD or the camera.
git clone https://github.com/edzzztech/Amy-AI.git cd Amy-AI pip install -r requirements.txt
Pull the local models
Install Ollama, then pull a chat model, a vision model and an embedding model. Amy can start the Ollama server herself on launch.
ollama pull llama3.2 # conversation + reasoning ollama pull llava # screen and camera vision ollama pull nomic-embed-text # memory search
Fill in your config
Open amy_config.json and replace the placeholders. Everything is optional except the Ollama block — skip any integration you don't want and that feature simply stays quiet.
"assistant": {
"wake_word": "amy",
"user_title": "Sir",
"default_city": "London",
"neural_voice": "en-GB-SoniaNeural"
}Say hello
She greets you on launch, then waits for the wake word. Say “Amy, what can you do?” to hear the current skill list.
python amy.py
Worth knowing about
amy_config.json holds every key and tunable — nothing is hard-coded in the source.
These are the settings most worth changing first.
| Setting | Default | What it does |
|---|---|---|
assistant.wake_word | amy | What she answers to. Pick something that doesn't collide with your normal speech. |
assistant.neural_voice | en-GB-SoniaNeural | Any Microsoft Edge neural voice. Free, no key, and far better than system TTS. |
assistant.barge_in | voice | How to interrupt her mid-reply. voice cuts her off as soon as you start talking. |
assistant.automation_approval | risky | When to ask before acting. risky gates destructive steps only; all gates everything. |
assistant.allow_shell_commands | false | Off by default, deliberately. Turn it on only if you want her running shell commands. |
ollama.auto_select | true | Routes each request to the fast, reasoning, code, CAD or vision model automatically. |
ollama.auto_start | true | Starts the Ollama server on launch if it isn't already running. |
proactive.level | normal | How often she speaks up unprompted, bounded by quiet_hours and max_per_hour. |
camera.start_on_launch | false | The webcam stays off until you ask for it. Gestures need it running. |
ui.theme | arc | Window theme, plus which controls appear in the bottom bar. |
Amy for Android, running on the device
A second build that runs on the phone itself rather than remote-controlling your computer — so it still works with the PC switched off. Same personality, same orb, same memory format.
Always listening
Say “Amy” and she answers. A conversation window after each exchange means follow-ups don’t need her name again.
On-device model
A small local model with GPU acceleration. With Gemma 3n she can see, too: photos from her camera and pictures you attach appear in the chat with her answer. Conversation stays on the phone and works in aeroplane mode.
Drives your apps
Opens apps, taps by name, types into fields and reads the screen through Android’s accessibility service. Root is optional and only adds to it.
A floating orb
The same mark hovering over everything else. Drag it anywhere, tap to open her, long-press to put her to sleep.
Phone controls
Timers, alarms, the torch and media volume. The time, the date and the battery are read from the phone, not guessed by the model.
Looks things up
“Google…” or “look up…” searches the web, and questions about news, prices, scores or the weather do it by themselves. She names her sources. Only the search words leave the phone.
Your files and your PC
Attach a text or Word document and ask about it. Pair with Amy on your computer, then say “on my PC…” to send it a command over your own network.
Installing it
There’s no Play Store build — you build it from source. About twenty minutes, most of it waiting for downloads.
Open the project
Install Android Studio, then open the android/ folder — not the repository root, which would confuse Gradle. Let it sync; the first run pulls down several gigabytes.
Put the phone in developer mode
Tap Build number seven times under Settings → About phone, then turn on USB debugging in Developer options. Plug the phone in, accept the prompt, pick it in the device dropdown and press Run.
Give her a model
For talking, Gemma 3 1B is about 0.6 GB. For her to see as well, Gemma 3n E2B is 3.7 GB — take gemma-3n-E2B-it-int4.litertlm. Both are gated, so accept the Gemma licence on Hugging Face first. Download it on the phone, then tap Attach in Amy and pick the file: she copies it into place and loads it. Or push it from a computer with adb (in PowerShell, Android Studio’s copy is at $env:LOCALAPPDATA\Android\Sdk\platform-tools\adb.exe):
adb push gemma-3n-E2B-it-int4.litertlm /sdcard/Android/data/io.github.edzzztech.amy/files/
A .task or .litertlm file in that folder is picked up whatever it’s called.
Grant what she needs
Allow the microphone and notifications when asked. For app control, turn on Settings → Accessibility → Amy — Android only lets an app open other apps from the background with that, or Display over other apps, switched on. Set her battery usage to Unrestricted, or Android will quietly kill the listening service after a while.
Common questions about running AI locally
What people usually want to know before installing an offline assistant.
Is Amy AI free?
Yes. The code is free to use, change and share for any noncommercial purpose under the PolyForm Noncommercial licence. There are no API keys, no subscription and no per-request cost, because the models run on your own hardware.
Does it work completely offline?
The assistant itself does. Speech recognition, the language model and the vision model all run locally, so conversation, screen reading and CAD work without a connection. Optional extras that are inherently online — weather, web research, Spotify — need one.
How is this different from Alexa, Siri or ChatGPT?
Those send your voice to a company's servers and can't touch your machine. Amy runs on your PC, sees your screen, opens and drives your applications, and keeps every transcript on your own disk.
What hardware do I need to run a local LLM?
Anything that can host your chosen Ollama model. A small model like llama3.2 runs on 8 GB of RAM; vision and reasoning models want more, and a GPU makes replies noticeably faster. Heavy parts load only when used.
Is any of my data sent anywhere?
Inference goes to localhost. Email, messaging and smart-home integrations are opt-in and only contact services you configure yourself. Credentials stay in your own config file on your machine.
Can it really control my computer?
Yes — it finds and drives real applications through Windows UI automation. Anything classed as risky pauses for your approval first, and shell command execution is off by default.
Is there a phone app?
Yes — an Android build that runs the assistant on the handset itself, so it works with your PC off. You build it from source; there’s no Play Store release.
Does it run on Mac or Linux?
Not yet. The automation layer uses Windows-specific APIs, so Amy currently targets Windows 10 and 11.
Can I use more than one camera?
Yes. Add as many as you like, give each a name, and switch by voice — “scan for cameras”, then “switch to the bench camera”.
Have a look at the source
One Python file, no services, no accounts. Clone it, point it at a local model, and talk to your computer.