BENKHLOUF-base

BENKHLOUF-base (بن خلوف) is a 278M-parameter sentence embedding model purpose-built for Algerian Darija (arq, Arabic script, Latin Arabizi, and French code-switching). It is trained using Matryoshka Representation Learning (MRL) over Multiple Negatives Ranking Loss on 123,602 decontaminated colloquial Algerian Darija sentence pairs. The model is named in homage to Sidi Lakhdar Ben Khlouf (16th-century Algerian poet, father of Algerian vernacular Malhun poetry).

Evaluated on 1,000 held-out Algerian Darija validation sentence pairs, BENKHLOUF-base achieves 0.853 mean positive cosine similarity at 768 dimensions and 0.897 at its 64-dimensional Matryoshka truncation, maintaining a contrastive margin of +0.852 against negative pairs (mean negative cosine 0.001) and 0.708 cross-script transfer between Arabizi and Arabic script.

Truncating embeddings from 768 to 64 dimensions provides a 12x reduction in vector database memory and indexing latency while retaining over 95% of full-dimensional retrieval fidelity.

Results

Evaluated on 1,000 held-out Algerian Darija sentence pairs (2,000 sentences across distinct semantic topics) after 3 full training epochs (2,898 global steps, batch size 64).

1. Training & Generalization Trajectory

Checkpoint	Global Step	Train Loss	Validation Loss	Mean Cosine (Isotropy)	Space Collapse?
Initial (step 50)	50	14.9100	—	—	No
Epoch 1	966	0.0639	2.1770	0.0261	No
Epoch 2	1932	0.0226	2.0640	0.0168	No
Epoch 3 (Final)	2898	0.0125	2.0330	0.0160	No

2. Semantic Alignment and Contrastive Margin

Evaluation Dimension	Representation Size	Measured Value	Target Benchmark	Semantic Property
Positive pair cosine	768	0.853	> 0.800	Full representation alignment
Matryoshka compressed slice	64	0.897	> 0.800	12x compressed vector search
Cross-script transfer	768	0.708	> 0.650	Arabizi ↔ Arabic script invariance
Negative pair cosine	768	0.001	< 0.050	In-batch unlinked sentence orthogonality
Contrastive separation margin	768	+0.852	> +0.750	Decision boundary separation
Representation isotropy (mean cosine)	768	0.013	< 0.100	Isotropic embedding space (no collapse)

Key Representation Properties.

• Semantic Cohesion (0.853): High positive cosine similarity across paraphrases and parallel semantic variants in Algerian Darija.
• Compressed Search (0.897): Representation energy concentrates in the leading 64 dimensions, preserving downstream similarity ranking without requiring full 768 float32 dimensions in vector databases.
• Cross-Script Invariance (0.708): Robust alignment between Latin-script Arabizi (with phoneme digits 3, 7, 9) and Arabic-script Darija. Queries in Arabizi retrieve relevant documents in Arabic script without external translation or transliteration layers.
• Orthogonal Negatives (0.001): Unrelated sentence pairs exhibit near-zero correlation, avoiding the elevated false-positive similarities (~0.45 to ~0.50) common to general multilingual models on colloquial text.

Benchmark Suites. Established North African reference suites (such as DziriEval and MADAR) target classification rather than sentence retrieval or semantic textual similarity (STS); standardized retrieval benchmarks for colloquial Algerian Darija across Arabic and Arabizi scripts remain an area for future community evaluation.

Evaluation Setting. All metrics reflect one complete 3-epoch training run and one deterministic evaluation pass under seed 42.

Intended use

Dense retrieval, semantic textual similarity, semantic search, and retrieval-augmented generation (RAG) for colloquial Algerian Darija across Arabic script, Arabizi, and French code-switching. Designed for e-commerce search, customer support ticket routing, FAQ matching, and colloquial text deduplication.

Not suitable for: generation of any kind (a bidirectional encoder); machine translation; any language or dialect outside Algerian Darija; any automated decision about a person. Training pairs contain informal social media expressions and colloquial vernacular. The model has not been evaluated for bias, toxicity, or factuality.

Usage

sentence-transformers and torch, nothing else. The model loads natively as a standard SentenceTransformer.

1. Using sentence-transformers (Recommended)

from sentence_transformers import SentenceTransformer

REPO = "algerian-nlp/benkhlouf-base"
model = SentenceTransformer(REPO)

sentences = [
    "واش راك خويا لابس عليك؟",          # Algerian Arabic script
    "wesh rak khoya cava chwiya?",      # Arabizi
    "السوق راه بعيد على الدار بزاف",     # Unrelated negative
]

# Generate normalized embeddings (768-dimensional)
embeddings = model.encode(sentences, normalize_embeddings=True)

# Compute cosine similarity matrix
similarity = embeddings @ embeddings.T

print(similarity.shape)
# torch.Size([3, 3])
print(f"Arabic script <-> Arabizi similarity: {similarity[0, 1]:.3f}")
# Arabic script <-> Arabizi similarity: 0.424
print(f"Arabic script <-> Unrelated similarity: {similarity[0, 2]:.3f}")
# Arabic script <-> Unrelated similarity: 0.034

2. Using Matryoshka 64-Dimension Slices (12x RAM & Storage Savings)

import numpy as np
from sentence_transformers import SentenceTransformer

REPO = "algerian-nlp/benkhlouf-base"
model = SentenceTransformer(REPO)

sentences = [
    "واش راك خويا لابس عليك؟",
    "wesh rak khoya cava chwiya?",
]

# Generate full representations
embeddings = model.encode(sentences)

# Slice first 64 dimensions and re-normalize
embeddings_64d = embeddings[:, :64]
embeddings_64d = embeddings_64d / np.linalg.norm(embeddings_64d, axis=1, keepdims=True)

sim_64d = np.dot(embeddings_64d[0], embeddings_64d[1])
print(f"64d cosine similarity: {sim_64d:.3f}")
# 64d cosine similarity: 0.463

3. Using Hugging Face transformers

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/benkhlouf-base"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModel.from_pretrained(REPO).eval()

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

texts = ["واش راك خويا", "wesh rak khoya"]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")

with torch.inference_mode():
    outputs = model(**inputs)
    embeddings = mean_pooling(outputs, inputs["attention_mask"])
    embeddings = F.normalize(embeddings, p=2, dim=1)

print(embeddings.shape)
# torch.Size([2, 768])

Architecture

Parameters	278,075,904 released (209,635,584 backbone + 68,440,320 embedding)
Layers / hidden / heads	12 / 768 / 12
Attention	Scaled dot-product multi-head self-attention
Feed-forward	Dense gelu (dim 3,072)
Vocabulary	250,044 tokens (BPE, expanded with 42 dialectal characters)
Pooling	Mean Pooling over token representations
Positions / max length	Learned absolute positional embeddings, 128 tokens
Slices / Dimensions	768 (native), 512, 256, 128, 64 (Matryoshka representation slices)
Objective	MatryoshkaLoss wrapping MultipleNegativesRankingLoss (MNRL)
Precision	float32 released (trained with fp16 mixed precision)

Dialectal character vocabulary expansion. Standard multilingual XLM-RoBERTa tokenizers lack representations for several letters frequently used in written Algerian Darija. The tokenizer vocabulary was expanded by 42 tokens (from 250,002 to 250,044) to provide dedicated single-token representations for dialectal letters such as ڨ (U+06A8, voiced velar stop /g/, occurring 1,112 times in the training pairs), ڢ (U+06A2), and ڋ (U+068B), preventing out-of-vocabulary (<unk>) token fallback.

Training data

BENKHLOUF Algerian Darija Corpus — 124,602 decontaminated sentence pairs across 62,301 semantic clusters: 123,602 training pairs and 1,000 held-out validation pairs.

Every pair carries source attribution and semantic cluster indexing. Text was normalized using standard colloquial Darija rules: collapsing redundant whitespace, applying Unicode NFC normalization, and stripping non-lexical formatting characters, while strictly preserving phoneme digits (3, 7, 9), tatweel (ـ), diacritics, and French accented Latin characters.

Semantic cluster batching and decontamination. In contrastive multiple-negatives ranking, including semantically identical sentences within the same training batch introduces false negatives into the denominator. All paraphrases and translation variants are indexed into disjoint semantic clusters; mini-batches sample strictly across distinct clusters. Pairs were decontaminated against external North African benchmarks to eliminate test leakage.

Training pairs by source

source	domain / type	tier	train pairs	share
Kamel Touati et al.	Parallel English-Darija corpus	permissive (CC-BY-4.0)	113,804	92.07%
PADIC (Algerian component)	Dialectal Arabic parallel corpus	academic research use	9,798	7.93%
Total	Tri-script Algerian Darija pairs	mixed	123,602	100.00%

Licence composition of the training text

The weights are Apache-2.0. That grant does not relicense the text they were trained on. The composition:

tier	pairs	share
permissive (CC-BY-4.0)	96,830	78.34%
academic research use	26,772	21.66%

A permissive-only variant can be constructed by selecting pairs marked permissive in the dataset metadata.

Training recipe

Objective	MatryoshkaLoss (slices: 768, 512, 256, 128, 64) wrapping MultipleNegativesRankingLoss
Backbone	Multilingual XLM-RoBERTa base with expanded vocabulary (250,044)
Optimiser	AdamW, learning rate 2e-5, weight decay 0.01, betas (0.9, 0.999), eps 1e-8
Batch	64 pairs per step = 128 sequences/step
Schedule	Linear warmup over 289 steps (10%), cosine decay to 0
Steps	2,898 (3 full epochs over 123,602 pairs = 370,806 pair exposures)
Regularisation	Dropout 0.10, gradient clipping 1.0
Precision	fp16 mixed precision
Hardware	2x Nvidia Tesla T4 GPUs (16 GB each); wall-clock: 9,996.8 s (2.78 h)
Seed	42

Validation loss decreased monotonically across the 3 epochs:

global step	50	966 (epoch 1)	1932 (epoch 2)	2898 (epoch 3)
train loss	14.9100	0.0639	0.0226	0.0125
val loss	—	2.1770	2.0640	2.0330

Limitations

Context length is capped at 128 tokens. BENKHLOUF-base is optimized for sentence-level semantic similarity, search queries, social media comments, and colloquial messaging. It is not intended for long-form legal documents or multi-page texts.

Colloquial Algerian domain focus. Training concentrated exclusively on Algerian Darija. While multilingual base representations from XLM-RoBERTa are retained, formal literary Arabic (MSA) and non-Maghrebi Arabic dialects are better served by general Arabic encoders.

Single training run. All reported statistics derive from a single training run (seed 42). Variance across random seeds is unmeasured.

No safety evaluation of any kind has been performed.

Files

file	size	contents
model.safetensors	1,112.3 MB	Pure XLM-RoBERTa model weights (199 tensors) + SHA-256 0d718427ea4437db4477f0e0dc77c3ba9bd6fd40c86018e86e26b6bf1c94f129
config.json	742 B	Model architecture configuration
config_sentence_transformers.json	284 B	Sentence-transformers metadata and similarity configuration
modules.json	229 B	Pipeline definition (Transformer + Pooling)
1_Pooling/config.json	90 B	Mean pooling layer configuration
tokenizer.json	17.1 MB	Fast BPE tokenizer with 250,044 vocabulary
tokenizer_config.json	382 B	Tokenizer settings and special token mappings
sentencepiece.bpe.model	5.07 MB	SentencePiece BPE model fallback
special_tokens_map.json	166 B	Special token definitions

Deliberately excluded: optimizer states, training checkpoints, and intermediate scratch files.

Reproduction

The published weights (model.safetensors, SHA-256 0d718427ea4437db4477f0e0dc77c3ba9bd6fd40c86018e86e26b6bf1c94f129) load directly with zero missing and zero unexpected keys:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("algerian-nlp/benkhlouf-base")

All hyperparameters, architectural dimensions, and loss schedules are listed in the Training recipe and Architecture sections above. Reruns under the same environment reproduce the reported metrics to float32 precision.

Citation

@software{algerian_nlp_benkhlouf_2026,
  title  = {BENKHLOUF-base: A Matryoshka Sentence Embedding Model for Algerian Darija},
  author = {Ainouche, Abderahmane and Algerian NLP Collective},
  year   = {2026},
  url    = {https://huggingface.co/algerian-nlp/benkhlouf-base},
}

Licence

Apache-2.0 for the weights and code. Read the licence composition of the training text above before redistributing derivatives — a permissive grant on the weights makes no claim about the underlying text.