In this note we use the State of the Union Address dataset from Kaggle to make some surprising (and some not so surprising) observations pertaining to the general timeline of American history, and the character and nature of the addresses themselves. Our main approach is using vector embeddings, such as BERT (DistilBERT) and GPT-2. While it is widely believed that BERT (and its variations) is most suitable for NLP classification tasks, we find out that GPT-2 in conjunction with nonlinear dimension reduction methods such as UMAP provide better separation and stronger clustering. This makes GPT-2 + UMAP an interesting alternative. In our case, no model fine-tuning is required, and the pre-trained out-of-the-box GPT-2 model is enough. We also used a fine-tuned DistilBERT model for classification (detecting which president delivered which address), with very good results (accuracy 93% - 95% depending on the run). All computations can be replicated by using the accompanying code on GitHub.

翻译：本文利用Kaggle上的“国情咨文数据集”（State of the Union Address），对美国历史总体时间线及演说本身的性质特征进行了一些令人惊讶（及部分意料之中）的观测。主要研究方法采用向量嵌入技术，如BERT（DistilBERT）和GPT-2。尽管普遍认为BERT（及其变体）最适用于自然语言处理分类任务，但我们发现GPT-2结合非线性降维方法（如UMAP）能够实现更优的数据分离与聚类效果。这使得GPT-2+UMAP成为一种具有潜力的替代方案。本案例中无需进行模型微调，预训练的原生GPT-2模型即可满足需求。同时，我们采用微调后的DistilBERT模型进行分类任务（识别总统与演说的对应关系），取得了极佳效果（准确率达93%-95%，因运行批次而异）。所有计算均可通过GitHub上附带的代码复现。