NunoMotaRicardo/llama-3.1-8b-financial-sentiment

LLaMA 3.1-8B Financial Sentiment Analysis (PEFT Adapters)

Fine-tuned PEFT (LoRA) adapters for meta-llama/Llama-3.1-8B-Instruct to perform 3-class sentiment classification on financial tweets.

⚠️ Important: This repository contains only the LoRA adapter weights, not the full model. You must have access to the base model meta-llama/Llama-3.1-8B-Instruct and load these adapters on top.

Model Description

This model uses QLoRA (Quantized Low-Rank Adaptation) to efficiently fine-tune LLaMA 3.1-8B for financial sentiment analysis. The adapters were trained on financial tweets to classify sentiment as Neutral, Bullish, or Bearish.

Key Features

• 🎯 PEFT Adapters Only: Repository contains only LoRA weights (~41M trainable parameters)
• 🔢 4-bit Quantization: Designed to work with NF4 4-bit quantized base model
• 💾 Memory Efficient: Runs on ~5-6GB VRAM with quantization
• 📊 Test Accuracy: 99.18% on 15% held-out test set
• 🚀 Inference Ready: Optimized for batch inference

Labels

Label ID	Sentiment	Description
0	Neutral	Objective or balanced market sentiment
1	Bullish	Positive market sentiment, optimistic
2	Bearish	Negative market sentiment, pessimistic

Training Details

Base Model

• Model: meta-llama/Llama-3.1-8B-Instruct
• License: Llama 3.1 Community License
• Compliance: This model is subject to the original Llama 3.1 license terms

PEFT Configuration

LoraConfig(
    r=16,                    # Rank of LoRA matrices
    lora_alpha=32,          # Scaling factor
    target_modules=[        # Attention + MLP layers
        "q_proj", "v_proj", "k_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_CLS"
)

Quantization Configuration

BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

Training Hyperparameters

• Epochs: 3
• Batch Size: 4 (gradient accumulation: 4, effective batch: 16)
• Learning Rate: 2e-4 with cosine scheduler
• Optimizer: Paged AdamW 8-bit
• Max Sequence Length: 128 tokens
• Training Samples: 38,119 (all available data, no validation split)
• Training Hardware: RTX 5060 Ti 16GB
• FP16: Enabled

Usage

Installation

pip install torch transformers peft bitsandbytes accelerate

Loading the Model

Important: You need access to the base Llama 3.1 model and must load these PEFT adapters on top.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
from peft import PeftModel, PeftConfig

# Configuration
model_id = "NunoMotaRicardo/llama-3.1-8b-financial-sentiment"
base_model = "meta-llama/Llama-3.1-8B-Instruct"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.pad_token_id = tokenizer.eos_token_id
tokenizer.padding_side = "left"

# Load base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

base = AutoModelForSequenceClassification.from_pretrained(
    base_model,
    num_labels=3,
    quantization_config=bnb_config,
    device_map="auto",
)
base.config.pad_token_id = tokenizer.pad_token_id

# Load PEFT adapters
model = PeftModel.from_pretrained(base, model_id)
model.eval()

Inference

# Single prediction
text = "$AAPL to the moon! 🚀 Great earnings, buying more calls!"

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=-1).item()

labels = {0: "Neutral", 1: "Bullish", 2: "Bearish"}
print(f"Sentiment: {labels[prediction]}")
# Output: Sentiment: Bullish

Batch Inference

tweets = [
    "$TSLA down 15%, brutal day. Cutting losses.",
    "$SPY consolidating, waiting for direction.",
    "$NVDA breaking out! Strong buy signal here!"
]

inputs = tokenizer(tweets, padding=True, truncation=True, max_length=128, return_tensors="pt")
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.argmax(outputs.logits, dim=-1).cpu().numpy()

for tweet, pred in zip(tweets, predictions):
    print(f"[{labels[pred]}] {tweet}")

Datasets

Training Data

• TimKoornstra/financial-tweets-sentiment: 38K real financial tweets with sentiment labels
• Note: TimKoornstra/synthetic-financial-tweets-sentiment 1.4M rows of synthetic data

Data Distribution

Neutral: ~33%
Bullish: ~43%
Bearish: ~24%

Limitations and Biases

• Domain Specific: Trained exclusively on financial tweets, may not generalize to other domains
• Twitter Language: Optimized for social media language with tickers, emojis, and slang
• Market Context: Sentiment labels reflect market sentiment, not general positivity/negativity
• Quantization: 4-bit quantization may introduce minor accuracy trade-offs vs full precision
• License Restrictions: Subject to Llama 3.1 Community License terms and usage restrictions

Ethical Considerations

• Not Financial Advice: This model's predictions should NOT be used as the sole basis for financial decisions
• Bias Awareness: May reflect biases present in financial Twitter discourse
• Market Manipulation: Should not be used to generate misleading sentiment signals

Citation

If you use this model, please cite:

@misc{llama31-financial-sentiment-peft,
  author = {NunoMotaRicardo},
  title = {LLaMA 3.1-8B Financial Sentiment Analysis (PEFT Adapters)},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/NunoMotaRicardo/llama-3.1-8b-financial-sentiment}
}

Acknowledgments

• Inspired by StephanAkkerman/FinTwitBERT-sentiment
• Based on Meta's Llama 3.1
• Datasets by TimKoornstra

Model Card Authors

NunoMotaRicardo

Model Card Contact

For questions or issues, please open an issue in the repository discussions.