BUCOLIN/berturk-milliyetner-ner

BERTurk Fine-Tuned on MilliyetNER for Named Entity Recognition

This repository provides a BERTurk model fine-tuned on the MilliyetNER dataset for Turkish Named Entity Recognition (NER).

The model was fine-tuned only on contemporary Turkish data from MilliyetNER. It was not fine-tuned on HisTR or Ruznamçe. Historical Turkish datasets were used only for evaluation to measure zero-shot cross-domain transfer performance.

Model Details

Item	Value
Base model	dbmdz/bert-base-turkish-cased
Architecture	BertForTokenClassification
Task	Named Entity Recognition
Training language	Turkish
Training domain	Contemporary Turkish news
Training dataset	MilliyetNER
Historical evaluation datasets	HisTR and Ruznamçe
Released seed	62
Number of labels	7

Label Set

The model uses BIO-formatted entity labels.

ID	Label	Description
0	B-LOC	Beginning of a location entity
1	B-ORG	Beginning of an organization entity
2	B-PER	Beginning of a person entity
3	I-LOC	Inside a location entity
4	I-ORG	Inside an organization entity
5	I-PER	Inside a person entity
6	O	Outside a named entity

Entity types:

• PER: Person
• LOC: Location
• ORG: Organization
• O: Outside a named entity

Training Strategy

The model was fine-tuned only on the MilliyetNER training set.

No Historical Turkish examples from HisTR or Ruznamçe were used during fine-tuning.

The evaluation design was:

1. Fine-tune BERTurk on the MilliyetNER training set.
2. Evaluate on the MilliyetNER development set.
3. Evaluate directly on the HisTR in-domain test set.
4. Evaluate directly on the Ruznamçe out-of-domain test set.

The HisTR and Ruznamçe results therefore represent zero-shot transfer from contemporary Turkish Named Entity Recognition to Historical Turkish Named Entity Recognition.

Training Configuration

Parameter	Value
Learning rate	5e-5
Number of epochs	10
Training batch size	16
Evaluation batch size	16
Weight decay	0.01
Maximum sequence length	256
Loss function	Cross entropy
Class weighting	None
CRF layer	No
Token-label alignment	First subtoken
Random seed	62

Run Selection

Three runs were trained using seeds 42, 52, and 62.

For this Hugging Face release, the run with the highest strict entity-level overall F1 score on the HisTR in-domain test set was selected, following the release criterion specified for this study.

Seed	HisTR In-Domain Overall F1
42	0.621
52	0.627
62	0.638

The released model is therefore the run trained with seed 62.

Evaluation

Evaluation was performed using strict entity-level matching. An entity was counted as correct only when both its span boundaries and entity type matched the gold annotation.

MilliyetNER Development Set

Overall Results

Metric	Score
Precision	0.720
Recall	0.680
F1	0.700

Entity-Level Results

Entity	Precision	Recall	F1
PER	0.520	0.480	0.500
LOC	0.8440	0.8762	0.8598
ORG	0.200	0.110	0.140

HisTR In-Domain Test Set

Overall Results

Metric	Score
Precision	0.6507
Recall	0.6264
F1	0.6380

Entity-Level Results

Entity	Precision	Recall	F1
PER	0.4365	0.4661	0.4508
LOC	0.8187	0.8827	0.8495
ORG	0.3130	0.0980	0.1493

Ruznamçe Out-of-Domain Test Set

Overall Results

Metric	Score
Precision	0.5189
Recall	0.5815
F1	0.5484

Entity-Level Results

Entity	Precision	Recall	F1
PER	0.4459	0.6407	0.5258
LOC	0.6818	0.4861	0.5676
ORG	0.7333	1.0000	0.8500

Usage

Using the Transformers Pipeline

Usage

Using the Transformers Pipeline

from transformers import pipeline

ner = pipeline(
    task="token-classification",
    model="BUCOLIN/berturk-milliyetner-ner",
    aggregation_strategy="simple",
)

text = "Mustafa Kemal Ankara'ya gitti."

predictions = ner(text)

for prediction in predictions:
    print(prediction)
### Loading the Model Directly

```python
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_id = "BUCOLIN/berturk-milliyetner-ner"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)

text = "Mustafa Kemal Ankara'ya gitti."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=256,
)

model.eval()

with torch.no_grad():
    outputs = model(**inputs)

prediction_ids = outputs.logits.argmax(dim=-1)[0].tolist()
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

for token, prediction_id in zip(tokens, prediction_ids):
    label = model.config.id2label[prediction_id]
    print(f"{token:<20} {label}")