https://hannody.github.io/tags/training-quality/Recent content in Training-Quality on Mohanad Abu-NaylaHugoen-USSat, 30 May 2026 00:00:00 +0000https://hannody.github.io/posts/ai/training/before-you-train-audit-tinystories/Sat, 30 May 2026 00:00:00 +0000https://hannody.github.io/posts/ai/training/before-you-train-audit-tinystories/<p>If you are building a small language model from scratch, TinyStories is usually one of the first datasets you reach for. I did too. Before touching tokenization and training, I ran a quick data audit on the parquet train split. That one step surfaced something worth sharing: a large number of exact duplicate stories sitting quietly in the data. This post is a practitioner-level writeup, not an accusation. TinyStories is still a solid dataset. The point is just this: verify your data pipeline the same way you verify your model pipeline.</p>