class Value:\n def __init__(self, data, children=()):\n self.data = data\n self.grad = 0\n self._backward = lambda: None\n self._prev = set(children)\n<\/code><\/pre>\n\n\n\nThis snippet is the core building block<\/em> of micrograd.<\/p>\n\n\n\n\nclass Value<\/code>: Each number in your computation isn\u2019t just a number anymore. It\u2019s a node in a graph, or neural network on a larger scale.<\/li>\n\n\n\ndata<\/code>: the raw number (like 2.5<\/code>).<\/li>\n\n\n\ngrad<\/code>: where the gradient (the derivative of the output in respect to the node) will be stored once you run “backpropagation”.<\/li>\n\n\n\n_backward<\/code>: a function placeholder that tells this node how to send its gradient backward to its parents when .backward()<\/code> is called.<\/li>\n\n\n\n_prev<\/code>: the set of \u201cchildren\u201d (really, the inputs that produced this node). This is what makes the whole thing a graph.<\/li>\n<\/ul>\n\n\n\nWhat I learned:<\/p>\n\n\n\n
\n- Graph thinking\n
\n- Computations are really a graph where each node tracks how it was created.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Backpropagation\n
\n-
.backward()<\/code> means gradients flow backward through every operation.<\/li>\n<\/ul>\n<\/li>\n\n\n\n- System over calculator\n
\n- This system is incredibly efficient, and building it can reap benefits that overshadow any basic calculator.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
Extra explanation (micrograd snippet):<\/strong><\/p>\n\n\n\n\n- Think of each
Value<\/code> as a spreadsheet cell that not only holds a number but also remembers the formula that produced it.<\/li>\n\n\n\n- During backprop,
_backward<\/code> is set to a tiny function for each operation (add, mul, tanh, etc.) so gradients can be pushed into grad<\/code> fields of the inputs.<\/li>\n\n\n\n_prev<\/code> lets you traverse the graph in reverse topological order to apply all those tiny _backward<\/code> functions.<\/li>\n<\/ul>\n\n\n\nGPT-2 Was Another Level<\/h3>\n\n\n\n
Micrograd in all of its glory prepared me for JUST THE BASICS. GPT-2 is pushed me into deep water (and a bit into insantity). The architecture looks simple on paper but was full of tricky details that took me a while to grasp.<\/p>\n\n\n\n
Attention Was Brutal<\/h4>\n\n\n\n
Understanding self-attention was the hardest part.<\/p>\n\n\n\n
class Head(nn.Module):\n def __init__(self, n_embd, head_size):\n super().__init__()\n self.key = nn.Linear(n_embd, head_size, bias=False)\n self.query = nn.Linear(n_embd, head_size, bias=False)\n self.value = nn.Linear(n_embd, head_size, bias=False)\n\n def forward(self, x):\n B, T, C = x.shape\n k = self.key(x)\n q = self.query(x)\n wei = q @ k.transpose(-2, -1) * (C ** -0.5)\n wei = torch.softmax(wei, dim=-1)\n v = self.value(x)\n return wei @ v\n<\/code><\/pre>\n\n\n\nWhat I learned:<\/strong><\/p>\n\n\n\n\n- Queries, keys, values: queries are \u201cwhat I\u2019m looking for,\u201d keys are \u201cwhat I have,\u201d values are \u201cthe content.\u201d<\/li>\n\n\n\n
- Shapes matter: batch size, sequence length, embedding dim; a single mismatch crashes the model.<\/li>\n\n\n\n
- Multi-heads: learn different relationships in parallel.<\/li>\n<\/ul>\n\n\n\n
Code explanation (attention head):<\/strong><\/p>\n\n\n\n\n- The head splits the input into three versions:<\/li>\n\n\n\n
- Query (Q)<\/strong> = what the token is looking for.<\/li>\n\n\n\n
- Key (K)<\/strong> = what the token has to offer.<\/li>\n\n\n\n
- Value (V)<\/strong> = the actual info.\n
\n- The math compares Qs with Ks to find relevance, then uses those scores to mix together Vs. The output is each token rewritten with context from the others.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
Positional Embeddings<\/h4>\n\n\n\n
Transformers aren\u2019t naturally sequential. Positional embeddings give them a sense of order, which turns out to be foundational.<\/p>\n\n\n\n
Code explanation (positional embeddings, conceptually):<\/strong><\/p>\n\n\n\n\n- Without positions, the model can\u2019t tell \u201cthe dog bit the man\u201d from \u201cthe man bit the dog.\u201d<\/li>\n\n\n\n
- Positional vectors are added to token embeddings so the model can learn order-sensitive patterns (like bigrams and syntax).<\/li>\n<\/ul>\n\n\n\n
Training Was Such a Freaking Grind<\/h3>\n\n\n\nfor step in range(max_iters):\n xb, yb = get_batch(\"train\")\n logits, loss = model(xb, yb)\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n if step % eval_interval == 0:\n val_loss = estimate_loss(\"val\")\n print(step, loss.item(), val_loss)\n<\/code><\/pre>\n\n\n\nWhat I learned:<\/p>\n\n\n\n
\n- Loss going down feels magical, even when it\u2019s just math.<\/li>\n\n\n\n
- Hyperparameters (LR, batch size, dropout) make or break runs.<\/li>\n\n\n\n
- Sometimes a single GPU and simpler tooling beat fighting with TPUs.<\/li>\n<\/ul>\n\n\n\n
Code explanation (training loop):<\/strong><\/p>\n\n\n\n\n- Grab a batch of inputs\/targets.<\/li>\n\n\n\n
- Run the model to get predictions and calculate loss.<\/li>\n\n\n\n
- Clear old gradients, backprop to compute new ones, and step the optimizer to update weights.<\/li>\n\n\n\n
- Every so often, check validation loss to see if the model is learning or overfitting.<\/li>\n<\/ul>\n\n\n\n
From Model to App: LittleGPT<\/h3>\n\n\n\n
The end result is a Streamlit app that:<\/p>\n\n\n\n
\n- Loads a small HF model locally (e.g., Qwen 0.6B) with device\/precision controls and
@st.cache_resource<\/code>.<\/li>\n\n\n\n- Lets you chat and optionally ground answers in your own uploaded notes via embeddings + FAISS RAG.<\/li>\n\n\n\n
- Supports quick LoRA fine\u2011tuning and simple evaluation.<\/li>\n<\/ul>\n\n\n\n
The Heartbeat: Generation<\/h4>\n\n\n\n
The core generation function uses plain transformers<\/code> under the hood. It prepares tensors, calls model.generate<\/code>, and decodes the output:<\/p>\n\n\n\ndef generate(\n model,\n tokenizer,\n prompt: str,\n max_new_tokens: int = 128,\n temperature: float = 0.0,\n top_p: float = 0.9,\n) -> str:\n \"\"\"Generate text from a prompt using lightweight decoding defaults.\"\"\"\n device = next(model.parameters()).device\n inputs = tokenizer(prompt, return_tensors=\"pt\").to(device)\n with torch.no_grad():\n temp = max(float(temperature), 0.0)\n do_sample = temp > 0\n sampling_temp = max(temp, 1e-5) if do_sample else 1.0\n output_ids = model.generate(\n **inputs,\n max_new_tokens=max_new_tokens,\n temperature=sampling_temp,\n top_p=min(max(float(top_p), 0.1), 1.0) if do_sample else 1.0,\n do_sample=do_sample,\n pad_token_id=tokenizer.pad_token_id,\n eos_token_id=tokenizer.eos_token_id,\n )\n generated = tokenizer.decode(output_ids[0], skip_special_tokens=True)\n if generated.startswith(prompt):\n return generated[len(prompt) :].strip()\n return generated.strip()\n<\/code><\/pre>\n\n\n\nCode explanation (generation):<\/strong><\/p>\n\n\n\n\n- Take the user\u2019s prompt, tokenize it, and feed it to the model.<\/li>\n\n\n\n
- If
temperature > 0<\/code>, the model samples more randomly. Lower = safer, higher = more creative.<\/li>\n\n\n\ntop_p<\/code> keeps sampling only from the most likely words.<\/li>\n\n\n\n- The model outputs tokens, we decode them back into text, and strip out the original prompt so only the continuation is shown.<\/li>\n<\/ul>\n\n\n\n
On the chat page, we compose a concise prompt template and call generate<\/code> with decoding controls sourced from the sidebar:<\/p>\n\n\n\nprompt_template = (\n \"You are Littlegpt, Connor's concise assistant. State your identity once at the start of a conversation and only repeat it if the user explicitly asks. Respond in at most four short sentences, without small talk, follow-up questions, or offers of extra help unless requested. Do not use emojis. Use the provided context when it helps.\\n\\n\"\n \"### User Instruction:\\n{instruction}\\n\\n### Context:\\n{context}\\n\\n### Answer:\".format(\n instruction=prompt.strip(),\n context=input_block.strip(),\n )\n)\n\nstart_time = time.time()\nwith st.chat_message(\"assistant\"):\n with st.spinner(\"Generating response...\"):\n raw_response = generate(\n model,\n tokenizer,\n prompt_template,\n max_new_tokens=int(st.session_state.get(\"max_new_tokens\", 128)),\n temperature=float(st.session_state.get(\"temperature\", 0.7)),\n top_p=float(st.session_state.get(\"top_p\", 0.9)),\n top_k=int(st.session_state.get(\"top_k\", 0)) or None,\n repetition_penalty=float(st.session_state.get(\"repetition_penalty\", 1.05)),\n no_repeat_ngram_size=int(st.session_state.get(\"no_repeat_ngram_size\", 3)) or None,\n )\n<\/code><\/pre>\n\n\n\nCode explanation (chat page call):<\/strong><\/p>\n\n\n\n\n- The app wraps the user\u2019s question and any retrieved context into a prompt template.<\/li>\n\n\n\n
- All the generation settings (tokens, temperature, top-p, etc.) come from the sidebar.<\/li>\n\n\n\n
- While it runs, Streamlit shows a spinner, then prints the model\u2019s answer with latency.<\/li>\n<\/ul>\n\n\n\n
Retrieval-Augmented Generation (RAG)<\/h3>\n\n\n\n
Uploads are chunked, embedded with a sentence-transformer, and searched via FAISS (if available). Top\u2011k snippets are stitched into the prompt.<\/p>\n\n\n\n
\n- Configure and add documents:<\/li>\n<\/ul>\n\n\n\n
@dataclass(frozen=True)\nclass RAGConfig:\n embed_model: str = \"sentence-transformers\/all-MiniLM-L6-v2\"\n chunk_size: int = 320\n overlap: int = 48\n<\/code><\/pre>\n\n\n\nCode explanation (RAGConfig):<\/strong><\/p>\n\n\n\n\n- Here is the code simplified:<\/strong><\/li>\n\n\n\n
embed_model<\/code>: the encoder that turns text into vectors.<\/li>\n\n\n\nchunk_size<\/code>: how big each text piece is.<\/li>\n\n\n\noverlap<\/code>: how much chunks overlap so no meaning is lost across boundaries.<\/li>\n<\/ul>\n\n\n\ndef add_document(self, doc_key: str, name: str, text: str) -> Tuple[int, int]:\n \"\"\"Add a document if it's not already cached. Returns (chunks_added, total_chunks).\"\"\"\n if doc_key in self._documents:\n return 0, self.chunk_count\n\n chunks = chunk_text(text, max_tokens=self.config.chunk_size, overlap=self.config.overlap)\n if not chunks:\n return 0, self.chunk_count\n\n embedder = self._embedding_model()\n embeddings = embedder.encode(\n chunks,\n batch_size=min(16, len(chunks)),\n convert_to_numpy=True,\n normalize_embeddings=True,\n )\n\n embeddings = _normalize(embeddings)\n\n start_idx = len(self._chunk_texts)\n new_indices = list(range(start_idx, start_idx + len(chunks)))\n\n self._chunk_texts.extend(chunks)\n self._chunk_sources.extend([name] * len(chunks))\n self._append_embeddings(embeddings)\n self._documents[doc_key] = RAGDocument(name=name, chunk_indices=new_indices)\n return len(chunks), self.chunk_count\n<\/code><\/pre>\n\n\n\nCode explanation (add_document):<\/strong><\/p>\n\n\n\n\n- Skip files you\u2019ve already added.<\/li>\n\n\n\n
- Split text into chunks, embed them, normalize the vectors, and save them with their source name.<\/li>\n\n\n\n
- This way the app can later search over them fast.<\/li>\n<\/ul>\n\n\n\n
def search(self, query: str, top_k: int = 3) -> List[Tuple[str, str, float]]:\n if not self._chunk_texts:\n return []\n\n embedder = self._embedding_model()\n query_embedding = embedder.encode(\n [query], convert_to_numpy=True, normalize_embeddings=True\n )\n query_embedding = _normalize(query_embedding)\n\n if faiss is not None and self._index is not None:\n scores, indices = self._index.search(query_embedding.astype(\"float32\"), top_k)\n results: List[Tuple[str, str, float]] = []\n if indices is not None and len(indices) and len(indices[0]):\n for idx, score in zip(indices[0], scores[0]):\n if idx is None or idx == -1:\n continue\n if not (0 <= int(idx) < len(self._chunk_texts)):\n continue\n j = int(idx)\n results.append((self._chunk_sources[j], self._chunk_texts[j], float(score)))\n return results\n\n if self._embeddings is None:\n return []\n\n sims = np.dot(self._embeddings, query_embedding.squeeze(0))\n if sims.ndim == 0:\n sims = np.array([float(sims)])\n top_indices = sims.argsort()[::-1][:top_k]\n return [\n (self._chunk_sources[idx], self._chunk_texts[idx], float(sims[idx]))\n for idx in top_indices\n if sims[idx] > 0\n ]\n<\/code><\/pre>\n\n\n\nCode explanation (search):<\/strong><\/p>\n\n\n\n\n- Turn the query into a vector and compare it with stored chunks.<\/li>\n\n\n\n
- If FAISS is available, use it for speed; otherwise just use dot products.<\/li>\n\n\n\n
- Return the best-matching snippets with their sources.<\/li>\n<\/ul>\n\n\n\n
context_chunks: List[tuple[str, str, float]] = []\nif rag_store.chunk_count:\n context_chunks = rag_store.search(prompt, top_k=rag_top_k)\n\nconversation_prefix = \"\\n\".join(\n f\"{msg['role'].capitalize()}: {msg['content']}\"\n for msg in st.session_state.chat_history[-6:-1]\n if msg[\"role\"] != \"system\"\n)\n\ncontext_sections: List[str] = []\nif conversation_prefix:\n context_sections.append(\"Conversation so far:\\n\" + conversation_prefix)\nif context_chunks:\n joined = \"\\n\\n\".join(f\"[{src}] {text}\" for src, text, _ in context_chunks)\n context_sections.append(\"Context documents:\\n\" + joined)\n<\/code><\/pre>\n\n\n\nCode explanation (wiring RAG):<\/strong><\/p>\n\n\n\n\n- Recent chat history is bundled with the retrieved document snippets.<\/li>\n\n\n\n
- They get stitched into a \u201cContext\u201d block that the model sees alongside the user\u2019s instruction.<\/li>\n<\/ul>\n\n\n\n
Under the hood, text chunking uses a simple token-approximate word stride:<\/p>\n\n\n\n
def chunk_text(text: str, max_tokens: int = 320, overlap: int = 48) -> List[str]:\n \"\"\"Split text into overlapping chunks sized ~max_tokens tokens (word-level).\"\"\"\n words = text.split()\n if not words:\n return []\n\n if max_tokens <= 0:\n return [text]\n\n stride = max(max_tokens - overlap, 1)\n chunks: List[str] = []\n for start in range(0, len(words), stride):\n chunk = \" \".join(words[start : start + max_tokens])\n if chunk:\n chunks.append(chunk)\n return chunks\n<\/code><\/pre>\n\n\n\nCode explanation (chunking):<\/strong><\/p>\n\n\n\n\n\n- Split the text into word windows of
max_tokens<\/code>, sliding forward by max_tokens - overlap<\/code>.<\/li>\n\n\n\n- Overlap makes sure info that straddles boundaries isn\u2019t lost.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
Fine\u2011Tuning with LoRA<\/h3>\n\n\n\n
Fine-tuning means taking a pre-trained model (like GPT-2) and teaching it extra patterns on a smaller, domain-specific dataset (like customer support chats, medical notes, or your own writing style). Instead of training from scratch, you \u201cnudge\u201d the model so it adapts quickly to your use-case. Below is an image of me in a Jupyter notebook using a v5e1 (high end Colab GPU) to train GPT-2. The table being printed shows step_loss<\/strong> (how wrong the model was at each step) and elapsed<\/strong> (how long training has run).<\/p>\n\n\n\n\n- Step 0: Loss was around 10.97<\/strong>.<\/li>\n\n\n\n
- By Step 30: Loss has dropped to around 7.64<\/strong>.<\/li>\n<\/ul>\n\n\n\n
That steady downward trend means the model is learning<\/strong>. It\u2019s adjusting weights to better predict tokens from your dataset. The Colab GPU makes this possible in minutes rather than days with something like my laptop.<\/p>\n\n\n\n![\"\"](\"https:\/\/wp.stgeorges.bc.ca\/connork\/wp-content\/uploads\/sites\/21\/2025\/08\/Screenshot-2025-09-21-at-5.30.13-PM-1024x733.png\")
<\/figure>\n\n\n\nYou can prototype instruction-tuning with LoRA right from the app. The training module prepares a Peft config over common projection layers and uses TRL\u2019s SFTTrainer<\/code>:<\/p>\n\n\n\nlora_config = LoraConfig(\n r=lora_r,\n lora_alpha=lora_alpha,\n lora_dropout=lora_dropout,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n target_modules=[\n \"q_proj\",\n \"k_proj\",\n \"v_proj\",\n \"o_proj\",\n \"gate_proj\",\n \"down_proj\",\n \"up_proj\",\n ],\n)\n<\/code><\/pre>\n\n\n\nCode explanation (LoRA config):<\/strong><\/p>\n\n\n\n\n- Instead of retraining all weights, LoRA slips in tiny adapters into projection layers.\n
\nr<\/code> (rank):<\/strong> You can think of it as the width of the adapter. Bigger r<\/code> = more capacity to learn, but also more compute.<\/li>\n\n\n\nalpha<\/code>:<\/strong> A scaling factor that adjusts how much influence the previously mentioned adapters have compared to the now frozen model weights.<\/li>\n\n\n\ndropout<\/code>:<\/strong> Randomly turns off parts of the adapter during training, which prevents the adapters from memorizing the training data and helps them generalize, which is better for model learning.<\/li>\n\n\n\n- target_modules:<\/strong> Lists exactly which parts of the transformer get these adapters. In this case, the projection layers (q_proj, k_proj, v_proj, etc.), which is the heart of self-attention.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
Training arguments and SFTTrainer<\/code> setup:<\/p>\n\n\n\ntraining_args = TrainingArguments(\n output_dir=tmp_dir,\n num_train_epochs=num_train_epochs,\n per_device_train_batch_size=batch_size,\n gradient_accumulation_steps=1,\n warmup_ratio=0.03,\n learning_rate=learning_rate,\n fp16=model.dtype == torch.float16,\n bf16=model.dtype == torch.bfloat16,\n logging_steps=max(1, len(dataset) \/\/ max(batch_size, 1)),\n save_strategy=\"no\",\n report_to=[],\n)\n\ntrainer = SFTTrainer(\n model=model,\n train_dataset=dataset,\n args=training_args,\n peft_config=lora_config,\n tokenizer=tokenizer,\n dataset_text_field=\"text\",\n max_seq_length=max_seq_length,\n packing=False,\n)\n<\/code><\/pre>\n\n\n\nCode explanation (training args + SFTTrainer):<\/strong><\/p>\n\n\n\n\n- Defines how to fine-tune: epochs, batch size, learning rate, precision (fp16\/bf16).<\/li>\n\n\n\n
- SFTTrainer runs supervised fine-tuning on text, with LoRA activated.<\/li>\n\n\n\n
packing=False<\/code> keeps each training example separate, making it simpler on my end.<\/li>\n<\/ul>\n\n\n\nQuick Evaluation<\/h3>\n\n\n\n
A small evaluation loop runs deterministic generations and computes exact match + a simple BLEU proxy:<\/p>\n\n\n\n
def _batched_generate(model, tokenizer, prompts: List[str], max_new_tokens: int) -> List[str]:\n device = next(model.parameters()).device\n encoded = tokenizer(prompts, return_tensors=\"pt\", padding=True, truncation=True).to(device)\n outputs = model.generate(\n **encoded,\n max_new_tokens=max_new_tokens,\n pad_token_id=tokenizer.pad_token_id,\n eos_token_id=tokenizer.eos_token_id,\n )\n decoded = tokenizer.batch_decode(outputs, skip_special_tokens=True)\n responses: List[str] = []\n for prompt, text in zip(prompts, decoded):\n if text.startswith(prompt):\n responses.append(text[len(prompt) :].strip())\n else:\n responses.append(text.strip())\n return responses\n<\/code><\/pre>\n\n\n\nCode explanation (evaluation generation):<\/strong><\/p>\n\n\n\n\n\n- Batch prompts, generate responses deterministically (no randomness), and clean up the outputs.<\/li>\n\n\n\n
- This gives stable, repeatable results for scoring.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
\n\n\n\nFlowchart<\/h3>\n\n\n\nHow to Read These Flowcharts<\/h5>\n\n\n\n
The app has three main flows:<\/p>\n\n\n\n
\n- Core Chat Flow<\/strong> – what happens when you open the app, type a prompt, and get an answer (with optional RAG if you\u2019ve uploaded notes).<\/li>\n\n\n\n
- LoRA Fine-Tuning Flow<\/strong> (left) – an optional path if you want to add adapters and specialize the model.<\/li>\n\n\n\n
- Evaluation Flow<\/strong> (right) – another optional path for testing the model on batches of prompts and scoring results.<\/li>\n<\/ol>\n\n\n\n\n
![\"\"](\"https:\/\/wp.stgeorges.bc.ca\/connork\/wp-content\/uploads\/sites\/21\/2025\/08\/Untitled-diagram-_-Mermaid-Chart-2025-09-25-213935-607x1024.png\")
<\/figure>\n<\/figure>\n\n\n\n\n![\"\"](\"https:\/\/wp.stgeorges.bc.ca\/connork\/wp-content\/uploads\/sites\/21\/2025\/08\/Untitled-diagram-_-Mermaid-Chart-2025-09-25-214148-439x1024.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/wp.stgeorges.bc.ca\/connork\/wp-content\/uploads\/sites\/21\/2025\/08\/Untitled-diagram-_-Mermaid-Chart-2025-09-25-214511-1-495x1024.png\")
<\/figure>\n<\/figure>\n\n\n\nTL;DR<\/h3>\n\n\n\n
Chat: <\/p>\n\n\n\n
Prompt – (optional RAG) – assemble final prompt – generate – decode\/return – show latency + tokens.<\/strong>
(LoRA fine-tune and Evaluation are optional side flows.)<\/em><\/p>\n\n\n\nCore chat flow<\/h4>\n\n\n\n\n- Open app – Sidebar
Pick device (CPU\/MPS\/CUDA), precision\/quantization, and gen params (max tokens, temperature, top-p).<\/li>\n\n\n\n - Load model – Cached base or LoRA adapter<\/li>\n\n\n\n
- Chat input – User types prompt<\/li>\n\n\n\n
- RAG branch (only if uploads exist)<\/li>\n<\/ol>\n\n\n\n
\n- Yes – Chunk – Embed – Build\/Use FAISS – Top-k search – produce context snippets.<\/li>\n\n\n\n
- No – Skip straight to prompt assembly.<\/li>\n<\/ul>\n\n\n\n
\n- Prompt assembly – Recent chat – Context snippets – Prompt template<\/li>\n\n\n\n
- Model inference – transformers.generate – Decode\/strip echoed prompt – Return answer<\/li>\n\n\n\n
- Metrics – Latency – Token counts<\/li>\n<\/ol>\n\n\n\n
\n\n\n\nLoRA fine-tuning flow (optional)<\/h4>\n\n\n\n
Start fine-tune – LoRA config (r, alpha, dropout, targets) – SFTTrainer + TrainingArguments – Train adapters – Save\/Load adapter – Model ready for chat<\/p>\n\n\n\n
Evaluation flow (optional)<\/h4>\n\n\n\n
Open Evaluate page – Batch prompts – Deterministic generate (no sampling) – Score (Exact Match + BLEU proxy) – Report results<\/strong><\/p>\n\n\n\n
\n\n\n\nEthics, Privacy & Sustainability<\/h4>\n\n\n\n
When I first built LittleGPT, I thought privacy was something I didn\u2019t need to worry about, since everything ran locally on my laptop. But I realized that good engineering practice is about more than just \u201cdoes it work for me.\u201d If someone else ran this app, they might upload sensitive files without thinking about how embeddings or indices are stored. That made me stop and think about how to design responsibly, even for a private tool.<\/p>\n\n\n\n
The RAG system is private by default. Documents are embedded in memory, and if they\u2019re cached, they sit in a local directory that can be cleared with a single command. Still, I started adding reminders in my write-up that people should avoid uploading files with personal data. If someone adapted this for public hosting, those warnings would be even more important.<\/p>\n\n\n\n
I also learned that bias doesn\u2019t go away just because the model is small or running offline. WikiText and C4 are public web datasets, and they contain stereotypes and skewed information. Any model trained on them inherits those patterns. Acknowledging that openly makes the project stronger, not weaker.<\/p>\n\n\n\n
The last piece was sustainability. Even a small Colab run uses electricity, and running dozens of experiments adds up. I began calculating the energy draw of my runs and realized that smaller models plus retrieval give you a better balance than just scaling up. That awareness made me more thoughtful about when to train, when to fine-tune, and when to rely on retrieval instead.<\/p>\n\n\n\n
Reproducible Evaluation<\/h4>\n\n\n\n
I wanted my evaluation to be something anyone could repeat. That meant fixing random seeds across Python, NumPy, and PyTorch, and also recording the environment details like which Python and PyTorch versions I was using, whether CUDA or MPS was available, and which model checkpoint I had loaded.<\/p>\n\n\n\n
For fairness, I used deterministic decoding: temperature at zero and top-p at one. That meant no random sampling and completely stable outputs. I also froze my validation set so it stayed constant.<\/p>\n\n\n\n
The three metrics I tracked were training loss, validation loss, and perplexity. Perplexity is simply exp(cross_entropy)<\/code>, but it feels more intuitive as a number because it tells you how many \u201cguesses\u201d the model is effectively making.<\/p>\n\n\n\nTo really see what worked, I also ran a couple of small ablations. Changing the number of attention heads or extending the context length changed performance in ways that were visible in the metrics. Summarizing them in a table gave me a clearer picture than just eyeballing the loss curve.<\/p>\n\n\n\n
Example table from my runs:<\/p>\n\n\n\nConfig<\/th> Params<\/th> Train Loss<\/th> Val Loss<\/th> Perplexity<\/th> Notes<\/th><\/tr><\/thead> Base<\/td> emb=256, heads=2, layers=4<\/td> \u2026<\/td> \u2026<\/td> \u2026<\/td> baseline<\/td><\/tr> Heads = 4<\/td> emb=256, heads=4, layers=4<\/td> \u2026<\/td> \u2026<\/td> \u2026<\/td> more heads<\/td><\/tr> Context = 512<\/td> emb=256, heads=2, layers=4<\/td> \u2026<\/td> \u2026<\/td> \u2026<\/td> longer sequences<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nAlongside this, I plotted a simple training vs. validation loss curve. It\u2019s one thing to see numbers in a table, but watching the validation curve flatten out while the training loss continues down is the clearest sign of overfitting.<\/p>\n\n\n\n
UX Tightening with CLI and Errors<\/h3>\n\n\n\n
Streamlit gave me a nice web interface, but I wanted something more reproducible and script-friendly. So I built a command-line entry point. Running python -m littlegpt.cli --help<\/code> shows everything in one place.<\/p>\n\n\n\nHere\u2019s what my --help<\/code> looked like:<\/strong><\/p>\n\n\n\n$ python -m littlegpt.cli --help\nusage: littlegpt.cli [-h] --prompt PROMPT [--model M] [--device cpu|mps|cuda]\n [--max_new_tokens N] [--temperature T] [--top_p P]\n [--rag_index PATH] [--top_k K] [--show_sources]\n [--seed 42]\n\nGenerate text locally with LittleGPT.\n\noptional arguments:\n -h, --help show this help message and exit\n --prompt PROMPT Text to continue\n --model M HF model id or local path (default: Qwen-0.5B)\n --device DEV cpu|mps|cuda (auto-detects)\n --max_new_tokens N Max tokens to generate (default: 128)\n --temperature T 0 = deterministic\n --top_p P nucleus sampling (default: 0.9)\n --rag_index PATH Optional FAISS index dir\n --top_k K Retrieval chunks to include (default: 3)\n --show_sources Print sources under the answer\n --seed S RNG seed for reproducibility\n<\/code><\/pre>\n\n\n\nI also included clear example commands in my post. One shows a fully deterministic run that I used when grading evaluation:<\/p>\n\n\n\n
python -m littlegpt.cli --prompt \"Explain positional embeddings in two sentences.\" --temperature 0\n<\/code><\/pre>\n\n\n\nAnd another demonstrates retrieval with citations:<\/p>\n\n\n\n
python -m littlegpt.cli --prompt \"Summarize my notes on ecosystems.\" \\\n --rag_index ~\/.littlegpt\/index --top_k 4 --show_sources\n<\/code><\/pre>\n\n\n\nFinally, I made sure the CLI spoke in plain English when things went wrong. Instead of PyTorch tracebacks, it now says things like \u201cCUDA not found, falling back to CPU\u201d or \u201cOut of memory at shape [B,T,C], try reducing context length.\u201d Those little touches made it feel much more polished.<\/p>\n\n\n\n
How to Run It<\/h4>\n\n\n\n
Local quickstart:<\/p>\n\n\n\n
python -m venv .venv && source .venv\/bin\/activate\npip install -r littlegpt\/requirements.txt\nstreamlit run littlegpt\/app.py\n<\/code><\/pre>\n\n\n\nCode explanation (local run):<\/strong><\/p>\n\n\n\n\n- Make a Python virtual environment, install requirements, run the Streamlit app.<\/li>\n\n\n\n
- It\u2019ll launch on localhost:8501.<\/li>\n<\/ul>\n\n\n\n
Run tests:<\/p>\n\n\n\n
pytest -q\n<\/code><\/pre>\n\n\n\nCode explanation (tests):<\/strong><\/p>\n\n\n\n\n- Run
pytest -q<\/code> to quickly check if things are working. If it returns no errors you’ll be good. In my experience, it’s typically you not installing proper packages if it doesn’t work.-q<\/code> runs pytest in quiet mode, great for smoke checks in CI or local validation.<\/li>\n<\/ul>\n\n\n\nExpose on a remote host during testing:<\/p>\n\n\n\n
streamlit run app.py --server.address 0.0.0.0 --server.port 7860\n<\/code><\/pre>\n\n\n\nCode explanation (expose app):<\/strong><\/p>\n\n\n\n\n- Running with
--server.address 0.0.0.0<\/code> makes the app visible from outside your machine, as long as the port is open.<\/li>\n<\/ul>\n\n\n\nDeploying on Hugging Face Spaces (Single Streamlit Space)<\/h4>\n\n\n\n
Follow the built-in guidance:<\/p>\n\n\n\n
huggingface-cli repo create <org>\/<space-name> --type=space --sdk=streamlit\n<\/code><\/pre>\n\n\n\nCode explanation (HF Spaces):<\/strong><\/p>\n\n\n\n\n- Create a new Space with Streamlit SDK, push your (which is really mine!) repo (with requirements.txt and app.py).<\/li>\n\n\n\n
- Spaces automatically runs the app online.<\/li>\n<\/ul>\n\n\n\n