saivamshiatukuri/qwen3.5-4b-decision-mind2web

Qwen3.5-4B decision model for browser steps (Mind2Web)

A LoRA adapter on Qwen/Qwen3.5-4B that answers two typed questions about a web-agent step in one forward pass, with no text generation: which of the candidate page elements should be acted on next (10 ranker candidates plus "none of the listed elements") and which operation (CLICK / TYPE / SELECT). It returns a probability over every option of every question, calibrated with a single temperature fitted on a held-out split, so an agent harness can act on the probability or escalate below a threshold.

It is the best of four decision-model designs we compared on this task ("Arm 3": LoRA on the language model, answers read from the logits at answer slots) in the study Decisions Without Generation: What Universal Decision Models Changed About Classification, and Where a Fine-Tuned 4B Model Still Wins (September 2026). The study compared purpose-tuned small classifiers with TypeSafe's Jev, a hosted universal decision model, on the decision surfaces Jev made visible: browser steps, tool selection, agent-run review, moderation, fact verification and knowledge.

Results

Official Mind2Web test splits, top-10 candidates from the Mind2Web paper's ranker rendered from cleaned HTML; a target outside the candidates is labelled "none of the listed elements" (27 % of the cross-task test). Accuracy in %, single seed; latency is batch-1 p50 on 100 real test rows (A100, adapter merged, bfloat16; Jev via API).

model	element (cross-task)	element (cross-website)	element (cross-domain)	operation	none recall	ECE (T)	latency
embedding + logistic regression (MiniLM, pairwise)	24.2	27.0	20.5	83.7	0.60	0.050	~0 (CPU)
zero-shot Qwen3.5-4B (same prompt, no training)	29.0	–	–	82.4	0.01	0.071	59 ms
frozen Qwen3.5-4B + linear head	25.8	–	–	82.5	0.03	0.078	≈ zero-shot
LoRA + [CLS]-per-option scorer (Arm 2b)	51.0	46.8	45.6	89.8	0.33	0.014	55 ms
this model: LoRA LM + slot logprobs (Arm 3)	53.2	50.3	48.1	89.3	0.36	0.024	62 ms
Jev (TypeSafe universal decision model, jev-1.13, API)	46.8	43.9	43.4	73.8	0.18	0.045	240 ms

Pooled over both questions on the cross-task test (n = 2,094 steps): accuracy 71.2 %, NLL 0.804, AUROC 0.84, selective accuracy at 90 % coverage 75.2 %. The same adapter, unchanged, was evaluated on the cross-website (n = 1,373) and cross-domain (n = 5,908) tests; it loses about five points out of distribution and stays ahead of every other model.

What the comparison says: one epoch on 6,787 labelled steps (about 20 minutes on one A100) gives +6 points over the universal model on element selection in distribution, +4 to +5 on unseen websites and domains, +15 on the operation, at a quarter of the API's latency. Served on a fully used A6000 the cost is about $1.9 per million decisions against $16.9 billed by the API for the same rows. The embedding floor's high none-recall is not a virtue; it predicts "none" because it cannot read the candidates.

The decision interface

state:      task + website + previous actions (text)
question 1: "Which of the candidate elements should be acted on next?"   enum, 10 candidates + "none of the listed elements"
question 2: "Which operation?"                                             enum, CLICK / TYPE / SELECT
output:     p(option) for every option of both questions, from one prefill

Prompt format (the Qwen chat template with thinking disabled; the assistant turn is teacher-forced to two placeholder lines and the model is read at the token before each ?):

<system>  You answer every question with a single letter, one per line, in order.
<user>    State:
          Task: Find all events taking place in New York City during the month of September.
          Website: seatgeek (Entertainment / Event)
          Previous actions:
          - [button]  Change Location -> CLICK

          Question 1: Which of the candidate elements should be acted on next?
          A) <button> Filter by Date
          B) <input placeholder="Search by city..." type="search">
          ...
          K) none of the listed elements

          Question 2: Which operation?
          A) CLICK
          B) TYPE
          C) SELECT
<assistant> 1) ?
            2) ?

The probability of an option is p(A) + p( A) at its slot, renormalised over the displayed options and divided by the fitted temperature (1.046, in decision_config.json). Option codes are single tokens in the Qwen tokenizer (A–Z, then two-letter codes).

Usage

With the study's code (pip install -r requirements.txt from the repository), which builds the prompt, reads the slots and applies the temperature:

from dm.decide import Decider

d = Decider("saivamshiatukuri/qwen3.5-4b-decision-mind2web", max_len=1536)   # temperature comes from decision_config.json
state = "Task: Find all events taking place in New York City during the month of September.\nWebsite: seatgeek (Entertainment / Event)\nPrevious actions:\n- (none yet)"
questions = [
    {"qid": "element", "type": "enum", "text": "Which of the candidate elements should be acted on next?",
     "options": ["<button> Filter by Date", "<button> Change Location", "<input name=\"search\" ...>", "...", "none of the listed elements"],
     "oos_index": 10},
    {"qid": "operation", "type": "enum", "text": "Which operation?", "options": ["CLICK", "TYPE", "SELECT"]},
]
for r in d.decide(state, questions):
    print(r["qid"], r["answer"], round(r["confidence"], 3))

With plain transformers + peft, load the adapter on the base model and read the logits yourself:

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "saivamshiatukuri/qwen3.5-4b-decision-mind2web").merge_and_unload()
# build the prompt as above; read model(**ids).logits at the token before each " ?"; p(option) = softmax over the option
# codes' bare and leading-space token ids, summed per option, then divided by T = 1.046 before the softmax.

Merge the adapter before serving (merge_and_unload): an unmerged adapter adds about 60 ms per decision on this backbone; merged, inference costs exactly what the base model costs.

Training

backbone	Qwen3.5-4B (hybrid Gated DeltaNet + attention), bfloat16, thinking disabled
adapter	LoRA r = 16, α = 32, dropout 0.05 on q/k/v/o, gate/up/down and the DeltaNet in_proj_qkv / in_proj_z / out_proj projections (~30M parameters)
data	Mind2Web train, 6,787 steps (cross-task split); 987 calibration steps carved from train; candidates = the paper ranker's top 10; 15 % of training targets are "none"
objective	0.7 · full-vocabulary cross-entropy at each answer slot + 0.3 · Brier over the displayed options
schedule	1 epoch, effective batch 32, AdamW, lr 1e-4, cosine with 3 % warm-up, gradient clipping 1.0, gradient checkpointing, max 1,536 tokens
selection	checkpoint by calibration-split NLL (0.764); temperature 1.046 fitted on the calibration split by NLL; test split never used for any choice
compute	about 20 minutes on one A100 80 GB