How to Code with Droids
If you can play Oregon Trail, you can code apps. Here’s a step-by-step guide for how artists, designers, devs, writers, and more can create software (apps, websites, tools, chrome extensions) using AI.
- Start with an idea…
- Create a plan…
- Execute in a smart way to build upon a series of micro-milestones..
- Expand, evolve, iterate and grow
- Do it all with AI.
Step 1: Start with an idea…
Record your thoughts… Talk it out. Start by talking about what you wish existed in the world. Use Apple notes, Granola, Zoom, or anything really. Just record your rambles.
Easy Trick → If you’re on a mac, open a new doc and hit F5 and it activates your mic and you can just talk and your words will appear as text in the doc. No other software needed.
What do you talk about? You don’t have to know how to actually solve any of the problems or if your idea is even possible. Pretend like you’re talking to a conversationally-permissive friend who lets you go on and on until you get to the core of your idea. Consider asking AI to rewrite and organize your ramble, or just copy and paste it. New to this? Try Granola, or even just open up ChatGPT and click on the little microphone button. Or my favorite… open up the ChatGPT mobile app, put on your headphones, and have a conversation. The idea here is to talk through what you want.
Step 2: 4 Minutes of Set-up.
Once you have a text transcript of what you are trying to do, we need to do four minutes of set-up. Go to Factory and sign up for a free account.
Don’t actually use factory on the web just yet. Just sign up. We’re going to use factory via your terminal.
Step 3: Open Terminal or Get VS Code
Open terminal on your mac, or if you do a little coding already, open VS Code or your favorite code editor. VS Code is free. Get it here https://code.visualstudio.com/download and then open a new terminal.
Step 4: Load in Factory
Type this into your terminal:
# Install droid
curl -fsSL https://app.factory.ai/cli | sh
# Navigate to your project
cd /path/to/your/project
# Start interactive session
droidBOOOOOM !!
That’s it! Just type the curl command and then droid!! and the whole thing launches. In the future, all you’ll need to do is type droid.
From here you literally type in commands into the text box like you were playing “Oregon Trail” or some other text-adventure game.
To see some options, hit the questions mark
# hit the question mark to see options
?
The most important ones are to change the model (as of Sept 20, 2025, I recommend switching to gpt-5, but feel free to do whatever you want).
To switch the model, first type /
# to switch the model, type slash
/
And then use the arrow keys to navigate up and down to get to the model and then hit return, and then choose the model you want.
Now we’re ready to code.
ALMOST
The secret to good coding is good planning, so first we need to make a plan. But that’s hard, so to do that well, we’ll ask the droid.
We first need to shift to the Spec mode to put everything into planning mode. This sounds hard, but it’s easy once you’ve done it once or twice. So just trust me for the next 15 seconds and we’ll get you there…
Hit shift-tab and watch the “mode” cycle through from coding to “spec”
Think of it as a “keyboard short cut” to switching menu items. So hit shift and tab at the same time twice and BOOOOM we’ve cycled to the magical “spec” mode of droid. You can tell if you did it correctly if you see Spec above the command line.
Step 5 Make a Plan.
- Ask the Droid in Spec Mode to make a detailed, step-by-step plan using your transcript from your brainstorming session.
- Ask the Droid to clarify anything that is contradictory or unclear.
- Ask it for suggestions to improve on your idea.
- Ask it to write a README and NEXTSTEPS docs that you will give to the Coding Droid in a minute to follow.
- Tell it to follow the engineering best practices for AI outlined by Buckhouse in his “Coding Backwards” article.
Here’s a sample prompt for the Droid in Spec Mode.
Prompt for Droid in SPec ModePlease take this transcript and generate a PRD, a PROJECT_PLAN, a NEXTSTEPS doc, a DESIGN_GUIDELINES doc, and a README. Put all of these docs into a folder called ./Docs Follow the engineering for AI best practices outlined by Buckhouse in his article "Coding Backwards" found here: https://buckhouse.medium.com/coding-backwards-f498805e96c1 the main point of the articles is to write the readme before you begin so that you can envision what success looks like, and then make a plan that follows a progressive build-and-test approach, so that you always have working code and make many, frequent, smart iterations to build upon code that works. Start small with something works. Choose each next steps to build this function, feature or test that will give you the most helpful information you need to be able to make progress towards the next milestone and is also easiest to debug. At every step, include robust logging for debugging. Design your logging so that the Coding Droid will get the information it needs to solve problems. At each step include a to-do to update all of the docs, especially the PROJECT_PLAN and NEXTSTEPS docs. Please note in the docs that I will occasionally go in an update the DESIGN_GUIDELINES to reflect design preferences. Please review the DESIGN_GUIDELINES when revising plans or next steps. Ask me for preferences for infrastructure, but for now, use Supabase and Vercel and GoogleAuth for all web apps. The documents you write will be read by Factory's Coding Droid, so write them so that the Coding Droid can best code the app. Think hard and write cogent plans. This product will be built by Factory.ai's Coding Droid, not by people, so make your plans in such a way that reflect how the Coding Droid best likes to work. Ask me any questions to get started. Attached is the transcript of my product brainstorm.
Step 6: Let’s Code
The Droid in Spec Mode will kick out a plan and write the docs to ./Docs. Now that we have a plan, Let’s code!
Hit shift & tab again to get into coding mode. Here I’ve selected “Medium Trust” so that it runs all the basic commands without asking me questions, but does ask before doing risky stuff.
Now we need to tell it to go do the work.
Here’s the prompt box, click on that little toolbox to get access to additional tools.
Prompt for Code DroidTake the plan created while in Spec mode and start to code. In this repo you'll find a ./Docs folder that contains all my project planning documents written while in Spec mode. Read through all the docs and then get started coding the first step in the project plan. This project has many, frequent "micro-milestones" that we'll hit along the way. Ask me any questions you have so that you can get coding.
Along the way you can feed it screen shots from Figma, update the design docs describing what you like and don’t like, and give it aesthetic directions, where you play the role of “hovering art director” or encouraging coach.
Factory will help you connect to Vercel, Supabase Google Auth, or anything else you need. Get stuck? Ask for help. Don’t know how to get API keys or save environment variables? Just ask and ask how to do it safely.
Step 7: Security Reviews
After each micro-milestone, ask Factory for a security review to make sure you haven’t done anything dangerous like check in your passwords, private keys (secret codes used to access APIs) or accidentally opened yourself up to malicious actors.
Prompt for Security Check:
We've just We've just finished a micro-milestone in our project plan.
Please do a quick security review to make sure we haven’t done anything
dangerous like check in my passwords, private keys, or accidentally
opened up the app to malicious input. If you're not sure if something
is safe, use your web browsing tools to double check.Step 8: Iterate
Now that you have the basic flow, keep iterating and expanding to get what you want. As you start to develop your own style, you can do things like create a stock pile of UI components that you like and instruct the Droid to use them as the basis for creating new UI components. You can also periodically ask it refactor, clean-up, to streamline your code. An important step to remember is to always ask the Droid to update the docs after they finish a task, otherwise, you’ll spend a lot of time “catching the AI up to where you are on the project” or correcting bad assumptions.
Problem Solving Guide:
- Problem: Factory gets sluggish. Solution: start a new session. Command-C to quit. Then type droid again.
- Problem: Factory keeps making the same mistake. Solution: Ask it to show you what files it’s stored in it’s memory. It’s likely holding on to an old idea it thinks is true. Or there’s something out of date in your ./Docs.
- Problem: I don’t understand why it’s doing something. Or even what it’s doing. Solution: Ask the Droid to explain it to you. It’s honestly my favorite part of building with Factory, learning from the Droids how to do something interesting.
It’s a really fun way to build. Here’s an example of a finished project: the SDL Learning Library.
SDL Learning Library—hundreds of tutorials, like Netflix for Builders…
It was all written in Factory. Supabase backend, CRUD interface for adding new videos, Vercel for hosting the code. GoogleAuth for the admin interface. Resend for auto-generated emails. Factory walked me through all of this.
Curious for more?
Subscribe to my newsletter for more on hands-on help and interesting stories as I explore the architecture of innovation and the nature of creativity.
https://jamesbuckhouse.substack.com/
❤
