x.com/rdolmedo_/status/2050665196340023765
1 correction found
Having pre-trained only on pre-1931 data!
This overstates the data cutoff. The talkie team says their pretraining corpus had temporal leakage, and the 13B model retained some post-1930 knowledge.
Full reasoning
The claim says the model was pre-trained only on pre-1931 data, but the model's creators explicitly say the pretraining corpus was not perfectly filtered for the 1930 cutoff.
In the official talkie announcement, the authors explain that preventing post-1930 material from entering the corpus was a key goal, but that their filtering "was not perfect." They give a concrete example: an earlier version of the model knew about the Roosevelt presidency and New Deal legislation, and they add that "talkie-1930-13b is additionally aware of some details related to World War II and the immediate postwar order."
If the model's pretraining corpus leaked enough post-1930 material for the model to know these later facts, then it was not literally pretrained only on pre-1931 data. A more accurate phrasing would be that it was intended to use only pre-1931 data or was primarily trained on such data, but the authors acknowledge some temporal leakage in pretraining.
1 source
- Introducing talkie: a 13B vintage language model from 1930
For talkie-1930, we developed a document-level n-gram-based anachronism classifier and used it to filter the pre-training corpus. However, this was not perfect... talkie-1930-13b is additionally aware of some details related to World War II and the immediate postwar order.