A practical learning journey through application logic, AI generation, and searching my own DevOps notes.
I use AI tools every day to learn, troubleshoot, and explore technical ideas. Eventually, I wanted to understand how an application actually produces those answers:
- Where does the knowledge come from?
- What happens between asking a question and receiving a response?
- How can I inspect that process when an answer is inaccurate?
Coming from a DevOps and AWS background, I chose a problem I understood: preparing for technical interviews and tracking what I needed to study. That became the KB AI DevOps Study Buddy.
1. Building the Learning Workflow
Interview preparation produces actionable signals about how we learn. A confident answer suggests mastery; an incomplete explanation highlights a gap. I wanted the application to use that information to suggest what to study next.
The early version brought together structured study plans, interview practice, basic answer evaluation, saved progress, weak-topic recommendations, and session summaries. Python handled the application logic, and Streamlit provided the user interface.
At this stage, behavior relied on static rules and pre-written question banks. That baseline made it easy to see where Generative AI was genuinely needed: maintaining a predefined collection of questions manually becomes repetitive and doesn’t scale.
2. Adding Generative Capabilities with Amazon Bedrock
To introduce dynamic scenarios, I integrated Amazon Bedrock, using Amazon Nova Lite for text generation.
A request could describe the specific topic and scenario:
“Create an AWS interview question about troubleshooting an Auto Scaling group that cannot launch additional instances.”
The application sent the prompt to Bedrock and displayed the generated question. Later, the integration was extended to AI-assisted answer evaluation and study-plan generation.
This separation of concerns was clean: the Python application managed the learning workflow and user state, while the foundation model supplied dynamic content within that workflow.
3. Connecting the Model to Personal DevOps Notes
The next requirement was straightforward in concept: answer questions using the DevOps notes inside my project.
Those Markdown files covered topics like AWS, Kubernetes, and Terraform. They were available to the application locally, but invoking Bedrock did not automatically give the model access to those files.
This introduced Retrieval-Augmented Generation (RAG):
| Component | Responsibility |
|---|---|
| Streamlit Interface | Collect the question and present the generated result. |
| Python Retrieval Logic | Search local DevOps notes and select candidate passages. |
| Prompt Construction | Combine the query, selected context, and instructions. |
| Amazon Bedrock & Nova Lite | Generate a grounded response using the supplied input. |
In this early iteration (RAG V0), retrieval lived entirely in custom application code rather than an external managed service. That gave me an end-to-end pipeline I could inspect and debug.
4. Making Retrieval Inspectable
A successful API response only confirms that the model answered. To judge technical validity, you must examine the raw material provided to the model in the prompt.
- Section Chunking: Notes were split by logical headers rather than arbitrary character counts. A section focused on Kubernetes Deployments is a far more relevant candidate for a replica-management question than a large file mixing unrelated concepts.
- Visibility: Exposing the retrieved passage and its file source directly made it possible to verify whether the source actually supported the model’s answer.
5. A Simple Example: Keyword Retrieval
Consider an illustrative question: “How do Kubernetes Deployments maintain replicas?”
After removing stopwords and normalizing tokens, a basic lexical retriever counts shared terms between the query and each chunk:
| Example Passage | Shared Query Terms | Score |
|---|---|---|
| “Kubernetes Deployments maintain the desired number of replicas.” | kubernetes, deployments, maintain, replicas | 4 |
| “Kubernetes Pods run application containers.” | kubernetes | 1 |
| “Terraform state maps resources to configuration.” | None | 0 |
import re
STOPWORDS = {"a", "an", "and", "do", "for", "how", "in", "is", "of", "the", "to"}
def keywords(text):
return set(re.findall(r"[a-z0-9]+", text.lower())) - STOPWORDS
def retrieve(question, chunks, top_k=3):
query_terms = keywords(question)
scored = [
(len(query_terms & keywords(chunk["text"])), chunk)
for chunk in chunks
]
ranked = sorted(scored, key=lambda item: item[0], reverse=True)
return [chunk for score, chunk in ranked if score > 0][:top_k]Lexical retrieval is clean and inspectable, but it hits immediate limits when phrasing changes. If a user asks:
“Why do extra copies of my app come back after I delete one?”
The query shares zero keywords with the target passage, yielding zero retrieved chunks despite the answer existing in the notes. This concrete gap establishes why vector embeddings and semantic search become necessary in later project phases.
6. Applying a DevOps Habit: Inspect the Inputs
In DevOps, an HTTP 200 response or an exit code 0 does not guarantee that a deployment is healthy—we inspect the inputs, logs, and telemetry against requirements.
For RAG systems, the exact same principle applies:
- Trace questions through the selected passages, prompt structure, and response.
- Maintain a repeatable evaluation set of questions to separate retrieval improvements from model phrasing variances.
- Explicitly design fallback behavior: when notes lack sufficient evidence, the application should state that context is missing rather than guessing.
Resources & Next Steps
- GitHub Repository: kb-ai-devops-study-buddy
- Part 1: Why I’m Building My Own AI DevOps Study Buddy
- Part 2: Before the AI: Building the First Version
- Part 3: The Study Buddy Finally Got AI
- Part 4: My AI Could Generate Answers, But Didn’t Know My Notes
- Part 5: My RAG Worked. But Was It Finding the Right Information?