The Five-Step AI Roadmap Everyone Shares — and What I Actually Run in Production
There is a graphic that makes the rounds every few weeks, and by now it has reached my feed in translated form, reposted by accounts I don’t follow. It lays out the path to working with large models in five steps: learn the fundamentals — Transformer, neural networks, attention — and stop as soon as you can read the diagrams; learn prompt engineering, which takes days; learn RAG so the model answers from your own data instead of making things up; learn agents, the biggest opportunity in the field; and then learn fine-tuning, where the advice is to skip pretraining entirely (that belongs to the big labs) and get comfortable with multimodal fine-tuning instead. There is a salary band attached to each step, and the band at the end is the reason the thing spreads.
I am not going to argue with the list. It is not wrong — it is a syllabus. And a syllabus is the kind of advice that behaves perfectly in a classroom and then dies on contact with a system, because it tells you what to learn and in what order, and never tells you which of those items you will still be doing years later when somebody is paying you for the result. I have been running AI workloads of my own for a while now — not research, not model training, but real services other people depend on. So here is the honest version of that graphic, checked against what is actually running on my hardware tonight.
Why this matters
The reason to distrust the roadmap is not that the items are wrong. It is that a roadmap tells you what to study, while employment asks what you can operate. Those are different skill sets, and the gap between them is where most people who “learned AI in five steps” stall: they can describe a RAG pipeline, but they have never had one break on them at 11pm in a way that made them understand retrieval, cost, ingestion and evaluation in a single evening.
The rest of this post maps each step to something I actually run, states plainly which steps I have never done, and gives you the order I would pick if I had to start again. Every claim below is measured on one desktop with an RTX 3060 (12 GB) and one NAS.
Step 1: Fundamentals — enough to be dangerous, then stop
The graphic’s framing here is closer to right than most engineers admit: learn the Transformer until the diagrams make sense, then move on. My version stopped earlier than that. I can hold a real conversation about attention, tokens, context windows, quantization schemes and the KV cache, because those are the levers I actually turn when a model has to fit into 12 GB of VRAM and answer fast enough to be useful in a chat window. I could not derive backpropagation on a whiteboard, and I would not trust a number I produced from memory.
What that costs me: nothing measurable so far, in systems that ship. What it would cost me: the day I train something. If your goal is “own the AI stack for a small business”, step 1 is a reading list, not a degree — and the honest reason to buy local hardware in the first place is not that a 12 GB card beats the frontier models, because it does not; it is what running a model yourself teaches you that an API never will. I wrote about that trade honestly in why I still bought a local GPU to run AI.
Step 2: Prompt engineering — the step with the lowest ceiling
“Days to learn” is accurate, and that is the problem with it. Prompting is the cheapest step to enter and the first one to get automated, so it is where enormous numbers of people stop and start describing themselves as AI engineers. The honest skill is not writing a better instruction; it is deciding when an instruction should not exist at all.
If a task has one correct answer, a deterministic input and a known output shape, then a script is faster, cheaper and cannot hallucinate. I have replaced agent prompts with forty lines of Python more than once, and wrote up the case for that. What survives on the prompt side is interface design: what the model is allowed to see, what shape its answer must take, how much of your own context you are willing to hand it, and — the part the graphic never mentions — how you would know the output got worse. That last item is evaluation, and in my experience it is the difference between a demo and a service.
Step 3: RAG — the hard half is everything except the retrieval
The one-line version everyone quotes is “give the model your data so it
stops lying”. I run exactly that: a self-hosted memory layer built from
mem0, LiteLLM and PostgreSQL 17 with pgvector, all on my own NAS, with
memories scoped to a user_id so one customer’s history cannot surface in
another’s answer —
the whole stack is written up here,
including the container layout and the fact that the API is two small HTTP
calls.
Three things the tutorial version of RAG skips, and each of them cost me real time:
- Ingestion is the boring half that breaks. Retrieval quality gets all the attention; getting the data in reliably is where the work is. The most recent input I pushed through my own speech-to-text stack was a 57-second video sitting on my desktop — it came back as text in about eight seconds on the same 12 GB card that already serves live dictation for my phone and laptop. That speed is what makes the stack usable instead of a demo, and the pipeline is documented end to end here.
- Byte-level mistakes look like model mistakes. My first attempt to
feed that same service through a workflow tool reported a clean 502
“conversion failed” from the audio layer. The real cause was a multipart
upload arriving without a
Content-Typeheader, which made the broker decode raw audio as UTF-8 text and destroy it before any of my code ran. Nothing about that bug is AI. It is exactly the kind of failure the five-step graphic cannot prepare you for. - Retrieval is not a model problem. When an answer is wrong, the useful question is whether the right chunk was ever retrieved, whether it survived chunking, and whether the model was given room to use it. Swapping to a bigger model has never once fixed a retrieval bug for me.
Underneath that entire layer there are two calls, which is worth seeing once:
POST http://127.0.0.1:20015/memories # store a fact or a conversation, tagged by user_id
GET http://127.0.0.1:20015/search # semantic search, scoped to the same user_idNothing exotic — which is the point. A memory layer you can describe in two lines is a memory layer you can debug, and the whole thing stays local.
Step 4: Agents — the opportunity is real, and the agent is the smallest part
I agree with the graphic’s ranking: this is where the work is. I disagree with what it implies by the word “agent”, because an agent is a model plus a loop plus tools — and the part that decides whether it survives contact with reality is the checking around it, not the model at the centre. I’ve written about this as loop engineering, and the finding I keep coming back to is uncomfortable: most of my scheduled automation has no AI in it at all, and those are precisely the jobs that have never needed babysitting. The model is the part that adds judgement. The rest of the system is what makes the judgement safe to trust.
Three things that are actually running here, so the claim is concrete rather than aspirational:
- A Telegram support bot for my own digital-goods shop that keeps long-term memory per customer, so a returning buyer is not a stranger on their second message. The memory layer it writes into is the same self-hosted mem0 stack from step 3.
- A scheduled job that manages discount settings in my Shopee seller account, which is the least glamorous useful agent I own: it reads the current state, decides, acts, and reports — and the entire value comes from the fact that it checks the result afterwards instead of assuming its click landed.
- A research run that compared twenty VPS providers in parallel by fanning the work out across subagents, which is a pattern worth learning even if you never build a “chatbot”: the write-up is here.
Step 5: Fine-tuning — the step I have never done, on purpose
The graphic says skip pretraining, which is correct, and then suggests learning multimodal fine-tuning. For anyone whose job looks like mine, I’d push back on the second half.
I have never fine-tuned a model. Not because I couldn’t, but because in the time I have been running these services, none of my failures have been model-behaviour failures. They were retrieval failures, licensing surprises, byte-level failures and operational failures — a job that reported success while the file it claimed to replace was untouched, a mount whose metadata lied about which file it had swapped, an API whose real model name differed from the one it advertised. A LoRA does not fix a schema drift.
Where I would fine-tune: one narrow, well-defined task; thousands of labelled examples rather than dozens; an inference bill or a latency ceiling that a smaller tuned model would genuinely fix; and the willingness to own an evaluation set, because a fine-tune you cannot measure is a regression you cannot see. And I would rent the GPU for it long before I bought a second card — the training run is a project, not a workstation purchase.
What I’d do differently
If I were starting from zero with that graphic in front of me, I would not walk it in numerical order. I would walk it in the order my problems arrived, and accept that the first real lesson will not come from a course:
- Treat step 1 as a reading list you finish in public documentation, not a curriculum. Enough theory to know why quantization and context limits hurt; no more until something forces you deeper.
- Spend a week on step 2, then immediately automate the parts of it that have one correct answer, and move the rest of your attention to evaluation — which the graphic omits entirely.
- Live in steps 3 and 4, because that is where nearly all employed AI work currently sits: wiring models to real data, and making the loop around them trustworthy.
- Keep step 5 in your back pocket until a specific problem names it. Fine-tuning is a technique with a precondition, not a rung on a ladder.
And notice the roadmap’s real omission, which is not a step but half the job: identity, storage, deployment, cost, monitoring, and who gets woken up when it breaks. That is the half that decides whether a team can use you, and it is the half this blog mostly writes about.
The result
Three of those five steps run in production on one desktop GPU and one NAS, and the honest scorecard is this:
- A 57-second video becomes text in about eight seconds, on the same 12 GB card that serves live dictation — and the speech server drops from 4,906 MiB to 1,321 MiB of VRAM when I toggle it idle, because a GPU that is always resident is a GPU I cannot use for anything else.
- The memory layer answers in two small HTTP calls, scoped per user, backed by PostgreSQL with pgvector on the NAS — mine, not a vendor’s.
- Most of the scheduled automation that keeps this stack alive contains no model at all, and it is the most reliable part of the system.
- Step 5 stays empty, and I would rather say that plainly than pad a roadmap. The gap is real, and it is a choice.
If the five-step graphic is the syllabus, this is the transcript. The syllabus tells you what to learn. The transcript is what a recruiter or a client can actually check.
Want an AI stack that runs in your business?
I run Hoelee Enterprise, a website design and development business for Malaysian SMEs — and the AI work above runs on the same infrastructure I build for clients. If you want a model answering from your data, on your hardware, without a per-seat subscription attached, I’d be glad to talk:
- 📱 WhatsApp: +60 12-797 2969
- 📧 Email: [email protected]
- 🌐 Website: hoelee.com
And if you are hiring for a platform, DevOps or AI-infrastructure role, the same channels reach me.
