I Built an AI Version of My Dad to Keep His Legacy Alive

My dad is a business consultant and lecturer who helped hundreds of companies grow. He’s now battling late-stage cancer, and this project became my way of preserving his voice, systems, and teachings.

This article covers how I built the tool technically — from transcription and retrieval to deployment.

💬 If you’re more interested in the story behind this build, why it matters, you can read the full personal story here.

🌐 Demo

🔧 Stack Overview

• Transcription: Baidu AI Cloud Speech-to-Text
• RAG Pipeline: LangChain
• Embeddings: OpenAI
• Voice Cloning: ElevenLabs (Instant Voice Clone)
• Frontend: Next.js + Tailwind
• Backend: FastAPI
• Deployment: EC2 + systemd services
• CI/CD: GitHub Actions

🛠️ The Build

At first, I tried transcribing 1.2 million characters of audio recordings, videos, everything manually through BaiduPan using my subscription. But switching between my phone and MacBook, clicking buttons nonstop, quickly became exhausting. Eventually, I wrote a script using Baidu AI Cloud to automate the transcription. It took me about a day to set up, mostly because I was lazy about reading the documentation, but once it worked, I transcribed all the audio and dumped everything into a single .txt file.

From there, I had two options according to my husband:

• Fine-tune a model with prompt/response pairs
• Build a RAG pipeline and retrieve context dynamically

I chose RAG — it was faster to iterate and didn’t require reformatting the entire corpus.

Quick aside: Fine-tuning teaches a model to replicate specific patterns.
RAG (Retrieval-Augmented Generation) keeps the base model unchanged and injects context from your own data — like documents or transcripts — at runtime.

To build the RAG, I used LangChain. I asked ChatGPT to generate a basic architecture, and it gave me a four-file structure to start from. I initially used a Chinese embedding model from HuggingFace and wrote a system prompt to reflect my dad’s tone and language.

It started working surprisingly well — it even sounded a bit like him. But the responses weren’t always accurate. I experimented with top_k, which controls how many documents are retrieved before the model answers. Lower values give more precision, while higher values provide more context but can add noise.

Eventually, I realized the problem wasn’t retrieval — it was the model’s small context window. So I switched to OpenAI’s embedding model, which turned out to be much faster and more accurate. Running HuggingFace on my EC2 server was also painfully slow, so this change made everything smoother.

Once the backend was functional, I moved on to the frontend.

💻 Frontend + Deployment

Thanks to my last project building a Spotify personal assistant, I had some experience. This time, I asked Claude to help me build a full Next.js frontend, instead of basic static HTML. The chatbot interface was simple, clean, and functional — complete with dark/light mode using Tailwind.

For deployment, I used an EC2 instance and managed everything myself. At first, I deployed the frontend and backend together, like I did with my last app. But it made debugging painful — when one service failed, everything failed.

So I split them into two systemd services, one for the backend and one for the frontend. This helped me debug each side independently and understand the pain points of real-world deployments. But I also started to understand the patterns and common failure points better.

🔊 Adding His Voice

I wanted people to hear him — not just read him.

So I trained a custom voice using ElevenLabs. I followed a single YouTube video, uploaded some voice clips, and used their instant voice feature. Their UI made it no-code, which was amazing.

After creating the voice, I used their API to generate audio from the chatbot’s responses. On the backend, I processed the OpenAI response and generated an .mp3 file. On the frontend, I added a simple audio player next to each message.

It doesn’t sound exactly like my dad — maybe 40% there. But it’s good enough for now. I added a backend toggle so audio only generates when a button is selected, since it’s expensive to run.

screenshot of the app with audio toggle

🔜 Next Steps

My parents have tested the chatbot, and my cousin (who now manages the business) is actively using it while documenting more of my dad’s teachings.

Here’s what I’m working on next:

• Continue refining prompts and chunking strategies
• Add more structured knowledge and QA patterns
• Explore integration with WeChat Work Bot for better client access

It may also be used to support upcoming books based on my dad’s systems — and to provide Q&A functionality for learners.

RAG isn’t perfect. It doesn’t offer the fine-tuned precision of a custom model, but it covers 80% of the use cases. For now, that’s enough.

🎓 What I Learned

I’m surprised I was able to pull this off in 2–3 days.

Most nights I was coding until 4 a.m., while my husband slept. I used up my GPT-o3 quota and had to rotate between Claude 4 and Gemini 2.5 Pro. Sometimes AI couldn’t fix things in one go — I had to rephrase my question three or four times to get a useful response.

There were frustrating moments. I sent the app to my parents and it didn’t work. I pulled an all-nighter to fix it. But I started to recognize patterns to help me debug: I learned to debug backend issues with /docs and frontend issues with the console.

Compared to my first app, this time my workflow had more structure. I set up GitHub Actions and versioned releases. I began to understand the relationship between backend and frontend — what needs to happen when, and what layer needs to be fixed.

When I first saw Tony Robbins’ AI, it felt like a massive, intimidating system. But when I broke it down to
transcription → embeddings → RAG → backend → frontend → deployment
it became completely manageable.

🪞Reflection

We often think of AI as something cold and corporate. But I built mine out of love — for my father, his teachings, and the people who still need them.

I’m proud of what he’s built. He started from nothing. His mission has always been clear: help businesses grow, help people lead better. It’s sad that he can’t continue doing what he loves. But I hope this AI lets people engage with his ideas, even in a small way.

It’s not emotional. It doesn’t replace him. It doesn’t capture the spark in his lectures or the way he lights up a room. But it holds his words. His systems. His wisdom.

And for me, it gave me a reason to revisit his work, to understand it deeply, to understand him better. It is a small, imperfect archive of love.