quran-live-stt-cpu / README.md
mohammed's picture
Update README.md
6fea9b8 verified
|
Raw History Blame Contribute Delete
10.1 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Quran Stt
emoji: ๐Ÿ“Š
colorFrom: pink
colorTo: purple
sdk: gradio
sdk_version: 6.16.0
python_version: '3.13'
app_file: app.py
pinned: false
license: mit
short_description: Real-time Quran recitation with FastConformer ASR
tags:
  - track:backyard
  - track:wood
  - sponsor:nvidia
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:sharing
  - achievement:fieldnotes

๐ŸŽ™๏ธ Quran Speech-to-Text

Real-time Quranic recitation transcription powered by a fine-tuned FastConformer model, achieving state-of-the-art accuracy on Arabic Quranic text with full diacritics.

๐Ÿ“น Watch Demo Video

๐Ÿ“ฑ Social Media Posts:

๐Ÿ”ฌ Model: fastconformer-quran-ar

This application uses a custom FastConformer model based on NVIDIA's architecture, fully fine-tuned on Quranic recitations from the tarteel-ai/everyayah dataset, which contains recitations from dozens of reciters across all 114 Surahs of the Quran.

Model Link: mohammed/fastconformer-quran-ar

๐Ÿ“Š Performance Metrics

  • Word Error Rate (WER): 0.14%
  • This means the model correctly recognizes 999 out of every 1000 words โ€” with complete diacritization (tashkeel)
  • ๐Ÿ† Ranked #2 on the Quranic ASR Leaderboard
  • State-of-the-art accuracy for Quranic recitation transcription

โš™๏ธ Training Methodology

The model was trained using a progressive unfreezing approach across five phases on an NVIDIA RTX 4070 Ti (12GB):

Phase Description Duration WER
Phase 1 Top 3 encoder layers 6h 5m 15.66%
Phase 2a Encoder layers 9-17 1h 15m 14.49%
Phase 2b Upper half encoder 10h 22m 0.18%
Phase 3a Full model finetune 2h 36m 8.71%
Phase 3b Full model (final) 4h 39m 0.14% โœ…

Total Training Time: ~25 hours WER Improvement: 99.1% reduction (from 15.66% โ†’ 0.14%)

This progressive approach allows for efficient training while achieving maximum accuracy by gradually unfreezing more layers.

๐ŸŽฏ Features

The model supports two operational modes:

  • Offline Transcription: Full transcription with maximum accuracy
  • Streaming Mode: Real-time transcription with low latency using cache-aware inference โ€” perfect for live recitation correction applications

๐Ÿš€ Built for Build Small Hackathon

This project is part of the Hugging Face Build Small Hackathon, focusing on practical AI solutions using efficient models.

๐Ÿ’ก Use Cases

  • Real-time Quran recitation correction
  • Quranic memorization assistance (Hifz)
  • Tajweed learning applications
  • Accessible Quran study tools
  • Quranic audio transcription and indexing

๐Ÿ› ๏ธ Technical Stack

  • Model Architecture: FastConformer (NVIDIA NeMo)
  • Framework: NeMo ASR
  • Interface: Gradio
  • Deployment: Hugging Face Spaces

๐Ÿ” Technical Implementation: Recitation Detection & Error Identification

This application implements a sophisticated real-time recitation verification system that goes beyond simple transcription. Here's how it works:

1. Quran Database & Text Normalization

Database Initialization (app.py:70-89)

  • Loads all 6,236 verses from the Quran using the quran-api
  • Maintains both original text (with full diacritics) and normalized text (without diacritics) for flexible matching

Arabic Text Normalization (app.py:48-59)

# Normalization process:
1. Remove all diacritics (tashkeel) - แธฅarakฤt, tanwฤซn, shaddah, etc.
2. Remove superscript alif (alif khanjariyah)
3. Normalize alif variations (ุฅ, ุฃ, ุข, ุง โ†’ ุง)
4. Normalize taa marbouta (ุฉ โ†’ ู‡)

This normalization enables accurate matching despite variations in pronunciation or reciter style while preserving the original diacritized text for display.

2. Intelligent Anchor Detection

Initial Verse Detection (app.py:109-162)

When a user starts reciting, the system must identify which verse they're reciting. This uses a position-based longest common subsequence algorithm:

  • Algorithm: difflib.SequenceMatcher finds the longest consecutive word match between transcribed audio and all verses in the database
  • Dynamic Thresholds based on recitation length:
    • Short (3-5 words): Requires 100% match from verse beginning
    • Medium (6-8 words): Requires โ‰ฅ85% coverage
    • Long (9+ words): Requires โ‰ฅ6 consecutive words with โ‰ฅ75% coverage
  • Position Preference: Prioritizes matches that start at the beginning of both the verse and the recitation
  • Stop-word Filtering: Ignores common Arabic particles (ุงู†ุŒ ู…ู†ุŒ ู…ุงุŒ ู„ุงุŒ ููŠุŒ ุนู„ู‰) to prevent false positives

Example: If user recites "ุจุณู… ุงู„ู„ู‡ ุงู„ุฑุญู…ู† ุงู„ุฑุญูŠู…", the system:

  1. Normalizes to "ุจุณู… ุงู„ู„ู‡ ุงู„ุฑุญู…ู† ุงู„ุฑุญูŠู…" (removes diacritics)
  2. Searches 6,236 verses for longest match
  3. Finds perfect 4-word match at start of Al-Fatiha (1:1)
  4. Sets anchor โ†’ expects verse 1:1

3. Real-Time Token-by-Token Alignment

Word-Level Error Detection (app.py:164-203)

Once anchored, the system performs streaming word-level comparison:

# Alignment process using edit distance opcodes:
1. Split canonical verse into tokens: ["ุจูุณู’ู…ู", "ูฑู„ู„ูŽู‘ู‡ู", "ูฑู„ุฑูŽู‘ุญู’ู…ูŽูฐู†ู", "ูฑู„ุฑูŽู‘ุญููŠู…ู"]
2. Split transcription into tokens: ["ุจุณู…", "ุงู„ู„ู‡", "ุงู„ุฑุญู…ู†", "ุงู„ุฑุญูŠู…"]
3. Normalize both for comparison
4. Use SequenceMatcher.get_opcodes() to detect:
   - 'equal': Correctly recited word โ†’ green highlighting
   - 'replace': Wrong word โ†’ red highlighting with underline
   - 'delete': Skipped word โ†’ red highlighting
   - 'insert': Extra word added โ†’ ignored in canonical display

Visual Feedback:

  • โœ… Correct words: Green text (<span class='correct-word'>)
  • โŒ Wrong words: Red bold underlined text (<span class='error-word'>)
  • โŒ Completely wrong verse: Red double-underlined (<span class='all-error-verse'>)

Streaming Display Constraint (app.py:192):

  • During active recitation: Only shows correctly matched words (prevents showing unrecited text)
  • After pause/completion: Shows all words including errors for review

4. Verse Completion & Auto-Advancement

Intelligent Verse Boundary Detection (app.py:332-348)

The system determines when a verse is complete using two methods:

Method 1: Token Exhaustion

canonical_tokens = "ุจูุณู’ู…ู ูฑู„ู„ูŽู‘ู‡ู ูฑู„ุฑูŽู‘ุญู’ู…ูŽูฐู†ู ูฑู„ุฑูŽู‘ุญููŠู…ู".split()  # 4 tokens
speech_tokens = transcribed_text.split()
is_completed = (last_matched_index >= len(canonical_tokens))

Method 2: Next-Verse Transition Detection

# If user starts reciting next verse while current verse incomplete:
1. Extract leftover tokens after last match
2. Compare with beginning of next verse (verse N+1)
3. If similarity โ‰ฅ75% โ†’ mark current verse complete, auto-advance

Upon Completion:

  1. Finalizes current verse with ayah number marker ๏ดฟูก๏ดพ
  2. Increments expected verse index
  3. Carries over leftover tokens to next verse
  4. If leftover tokens don't match next verse โ†’ marks as error

5. Continuous Streaming Pipeline

Real-Time Audio Processing (app.py:222-415)

Audio Stream โ†’ Accumulation โ†’ ASR Transcription โ†’ Alignment โ†’ Display
     โ†“              โ†“              โ†“                  โ†“           โ†“
  16kHz WAV    0.5s minimum   FastConformer    SequenceMatcher  HTML
                buffer        (streaming)      (live tracking)  update

Key Features:

  • Silence Detection: RMS < 0.005 triggers pause (2 chunks = ~0.4s)
  • Streaming ASR: Runs transcription every 0.5+ seconds on accumulated audio
  • Live Preview: Shows real-time alignment even before pause
  • Commit on Pause: Finalizes verse detection only during valid pauses
  • State Persistence: Maintains verse context, partial recitation, and history across chunks

6. Error Classification

The system distinguishes three error types:

Error Type Detection Method Visual Display
Wrong Word Token mismatch via replace opcode Red underlined word
Skipped Word Missing token via delete opcode Red underlined word in canonical position
Wrong Verse No anchor found or <75% next-verse match Entire verse in red double-underline

Performance Characteristics

  • Latency: ~200-500ms from speech to visual feedback (depending on chunk size)
  • Memory: Streams audio in chunks; clears after each pause
  • Accuracy: Inherits 0.14% WER from base model, enhanced by verse-constrained matching
  • Robustness: Handles pronunciation variations through normalization and fuzzy matching

This architecture enables the application to function as an intelligent Quran tutor that not only transcribes but actively guides users through sequential memorization with immediate, precise feedback.

๐Ÿ“ License

MIT License