NewsBERT 1800 1920 Temporal

NewsBERT_1800-1920-Temporal A fine-tuned version of , conditioned on each item's publication year as a continuous input. A single model was trained across the full 1800-1920 span of the original NewsBERT_1800-1920's dataset, with each training example's year injected as a sinusoidal (Fourier) feature added to the token embeddings. Nearby years produce nearby embeddings by construction, so the model can be queried at any year in-range. ## Architecture - : (BERT-base architecture) - : a small module maps a normalized year to a -dimensional vector via sinusoidal features at 24 log-spaced frequencies, spanning: - lowest frequency: 1 cycle over the full 120-year corpus span - highest frequency: 1 cycle per 5 years - This vector is added to every token's input embedding, then passed through a before entering the transformer stack (this re-normalization step was added after an earlier version without it let the time signal's magnitude dominate the token embeddings it was added to, which suppressed learning). You need the accompanying in this repo to load this model, as it is loadable via a plain call, since the time-injection logic is outside the standard BERT forward pass. ## Training data 9.28M items (, ) from the LwM and HMD collections (see ), spanning 1800-1920, split into overlapping 126-token windows (stride 96), i.e. 94.0M training windows. ## Training procedure - 1 epoch, batch size 256, linear LR decay, 500 warmup steps - Learning rate: for the base model's existing (pretrained) layers, and a higher rate of for the ContinuousTimeEmbedding module since that module is new and starts from scratch - bf16 mixed precision ### Hardware / environment This research utilised Queen Mary's Apocrita HPC facility, supported by QMUL Research-IT. - Single GPU per training job - GPU: NVIDIA A100-PCIE-40GB (40GB VRAM) - CUDA: 12.6 (as built into the version used) - Python 3.11.7 (GCC 12.2.0 build) - 2.11.0+cu126 - 5.12.1 ## Example usage ### Load the model and the companion module ### Basic functions to get top mask predictions and token probability ### Check changes in top mask prediction or in token probability Example outputs: ### Check against known historical events Visualize changes in probability of particular tokens and compare against historical events which should correspond to peak in probabilities (either one-off for a specific year or as the start of a continuous rise in probability thereafter, e.g. due to increased discourse around it in the newspapers). Example output: ## Known limitations - Trained for 1 epoch only. Treat results as preliminary. - No year/decade-reweighting was used for this specific run, so the model's competence will not be uniform across the full 1800-1920 span and will reflect whatever the skewed training corpus's year distribution is.