Before You Train, Audit: A TinyStories Case Study
If you are building a small language model from scratch, TinyStories is usually one of the first datasets you reach for. I did too. Before touching tokenization and training, I ran a quick data audit on the parquet train split. That one step surfaced something worth sharing: a large number of exact duplicate stories sitting quietly in the data. This post is a practitioner-level writeup, not an accusation. TinyStories is still a solid dataset. The point is just this: verify your data pipeline the same way you verify your model pipeline. ...