--- license: mit tags: - event-sequences - synthea - workshop --- # synthea-bert An encoder-only, BERT-style masked model over **sequences of health events**, trained on a synthetic [Synthea](https://github.com/synthetichealth/synthea) population. Teaching material for *Transformer Architectures for Computational Social Science* (ICSC 2026, Oxford). ## Do not load this with `AutoModelForMaskedLM` The model adds **Time2Vec** time encodings to the input embeddings. Loading the inner `BertForMaskedLM` directly skips them, and the model then returns confident nonsense while the loss still looks fine. Use the loader shipped in this repo: ```python import sys import pandas as pd from huggingface_hub import snapshot_download model_dir = snapshot_download("carlomarxx/synthea-bert") data_dir = snapshot_download("carlomarxx/synthea-workshop-data", repo_type="dataset") sys.path.insert(0, model_dir) # event_bert.py ships alongside the weights from event_bert import EventBertForMaskedLM vocabulary = pd.read_csv(f"{data_dir}/vocabulary.csv") model = EventBertForMaskedLM.from_pretrained( model_dir, expected_vocab_size=len(vocabulary) ) ``` ## Vocabulary 1036 tokens: 5 special, 46 background and the rest event types. The background tokens are a person's static attributes, placed at the head of the sequence following life2vec rather than bolted on downstream as features: 6 age bands, 2 sex, 6 race, 2 ethnicity, 5 marital status, and **25 equal-frequency income bins**. The vocabulary lives in [`carlomarxx/synthea-workshop-data`](https://huggingface.co/datasets/carlomarxx/synthea-workshop-data) and the two are only valid as a pair — `expected_vocab_size` asserts this. ## What it learned Synthea is a **rule-based simulator**, so structure recovered from it reflects generator rules, not epidemiology. Measured consequence: on scalar outcomes this model ties or loses to a tabular baseline of age plus record volume. On *next-event* prediction it wins decisively (0.619 top-1 against 0.303 for a bigram baseline), because the generator wrote care pathways — vaccination schedules, dental bundles, screening batteries — and pathways are what it learned. Published from [`carlomarxdk/workshop-transformers`](https://github.com/carlomarxdk/workshop-transformers) at commit `ffae814`.