--- title: Quran Stt emoji: ๐Ÿ“Š colorFrom: pink colorTo: purple sdk: gradio sdk_version: 6.16.0 python_version: '3.13' app_file: app.py pinned: false license: mit short_description: Real-time Quran recitation with FastConformer ASR tags: - track:backyard - track:wood - sponsor:nvidia - achievement:offgrid - achievement:welltuned - achievement:offbrand - achievement:sharing - achievement:fieldnotes --- # ๐ŸŽ™๏ธ Quran Speech-to-Text Real-time Quranic recitation transcription powered by a fine-tuned FastConformer model, achieving state-of-the-art accuracy on Arabic Quranic text with full diacritics. ๐Ÿ“น **[Watch Demo Video](https://youtu.be/EP0yBn_05S0?si=VkQRlY5U3MSHZr_3)** ๐Ÿ“ฑ **Social Media Posts:** - [Fine-tuned Model Announcement](https://www.linkedin.com/posts/mrabkoo_aepaesaeqaezaekaeuabraepaesaeraezaeyaet-aepaesaeyaeraepaejabraepaesaepaedaefaeuaepaehaey-share-7470200908539412481-8DWy/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABI_t9EBJu1_ZlYUFCc5a-WnllxkJhfoDqo) - [Gradio Space Application](https://www.linkedin.com/posts/mrabkoo_aepaesaeqaezaekaeuabraepaesaeraezaeyaet-aepaesaeyaeraepaejabraepaesaepaedaefaeuaepaehaey-ugcPost-7471921360408219648-Q5kM/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABI_t9EBJu1_ZlYUFCc5a-WnllxkJhfoDqo) ## ๐Ÿ”ฌ Model: fastconformer-quran-ar This application uses a custom FastConformer model based on NVIDIA's architecture, fully fine-tuned on Quranic recitations from the [tarteel-ai/everyayah](https://huggingface.co/datasets/tarteel-ai/everyayah) dataset, which contains recitations from dozens of reciters across all 114 Surahs of the Quran. **Model Link:** [mohammed/fastconformer-quran-ar](https://huggingface.co/mohammed/fastconformer-quran-ar) ## ๐Ÿ“Š Performance Metrics - **Word Error Rate (WER): 0.14%** - This means the model correctly recognizes **999 out of every 1000 words** โ€” with complete diacritization (tashkeel) - **๐Ÿ† Ranked #2 on the [Quranic ASR Leaderboard](https://huggingface.co/spaces/Muno459/quranic-asr-leaderboard)** - State-of-the-art accuracy for Quranic recitation transcription ## โš™๏ธ Training Methodology The model was trained using a **progressive unfreezing** approach across five phases on an NVIDIA RTX 4070 Ti (12GB): | Phase | Description | Duration | WER | |-------|-------------|----------|-----| | **Phase 1** | Top 3 encoder layers | 6h 5m | 15.66% | | **Phase 2a** | Encoder layers 9-17 | 1h 15m | 14.49% | | **Phase 2b** | Upper half encoder | 10h 22m | 0.18% | | **Phase 3a** | Full model finetune | 2h 36m | 8.71% | | **Phase 3b** | Full model (final) | 4h 39m | **0.14%** โœ… | **Total Training Time:** ~25 hours **WER Improvement:** 99.1% reduction (from 15.66% โ†’ 0.14%) This progressive approach allows for efficient training while achieving maximum accuracy by gradually unfreezing more layers. ## ๐ŸŽฏ Features The model supports two operational modes: - **Offline Transcription**: Full transcription with maximum accuracy - **Streaming Mode**: Real-time transcription with low latency using cache-aware inference โ€” perfect for live recitation correction applications ## ๐Ÿš€ Built for Build Small Hackathon This project is part of the [Hugging Face Build Small Hackathon](https://huggingface.co/build-small-hackathon), focusing on practical AI solutions using efficient models. ## ๐Ÿ’ก Use Cases - Real-time Quran recitation correction - Quranic memorization assistance (Hifz) - Tajweed learning applications - Accessible Quran study tools - Quranic audio transcription and indexing ## ๐Ÿ› ๏ธ Technical Stack - **Model Architecture**: FastConformer (NVIDIA NeMo) - **Framework**: NeMo ASR - **Interface**: Gradio - **Deployment**: Hugging Face Spaces ## ๐Ÿ” Technical Implementation: Recitation Detection & Error Identification This application implements a sophisticated real-time recitation verification system that goes beyond simple transcription. Here's how it works: ### 1. Quran Database & Text Normalization **Database Initialization** ([app.py:70-89](app.py#L70-L89)) - Loads all 6,236 verses from the Quran using the [quran-api](https://cdn.jsdelivr.net/gh/fawazahmed0/quran-api@1/editions/ara-quransimple.json) - Maintains both **original text** (with full diacritics) and **normalized text** (without diacritics) for flexible matching **Arabic Text Normalization** ([app.py:48-59](app.py#L48-L59)) ```python # Normalization process: 1. Remove all diacritics (tashkeel) - แธฅarakฤt, tanwฤซn, shaddah, etc. 2. Remove superscript alif (alif khanjariyah) 3. Normalize alif variations (ุฅ, ุฃ, ุข, ุง โ†’ ุง) 4. Normalize taa marbouta (ุฉ โ†’ ู‡) ``` This normalization enables accurate matching despite variations in pronunciation or reciter style while preserving the original diacritized text for display. ### 2. Intelligent Anchor Detection **Initial Verse Detection** ([app.py:109-162](app.py#L109-L162)) When a user starts reciting, the system must identify which verse they're reciting. This uses a **position-based longest common subsequence algorithm**: - **Algorithm**: `difflib.SequenceMatcher` finds the longest consecutive word match between transcribed audio and all verses in the database - **Dynamic Thresholds** based on recitation length: - **Short (3-5 words)**: Requires 100% match from verse beginning - **Medium (6-8 words)**: Requires โ‰ฅ85% coverage - **Long (9+ words)**: Requires โ‰ฅ6 consecutive words with โ‰ฅ75% coverage - **Position Preference**: Prioritizes matches that start at the beginning of both the verse and the recitation - **Stop-word Filtering**: Ignores common Arabic particles (ุงู†ุŒ ู…ู†ุŒ ู…ุงุŒ ู„ุงุŒ ููŠุŒ ุนู„ู‰) to prevent false positives **Example**: If user recites "ุจุณู… ุงู„ู„ู‡ ุงู„ุฑุญู…ู† ุงู„ุฑุญูŠู…", the system: 1. Normalizes to "ุจุณู… ุงู„ู„ู‡ ุงู„ุฑุญู…ู† ุงู„ุฑุญูŠู…" (removes diacritics) 2. Searches 6,236 verses for longest match 3. Finds perfect 4-word match at start of Al-Fatiha (1:1) 4. Sets anchor โ†’ expects verse 1:1 ### 3. Real-Time Token-by-Token Alignment **Word-Level Error Detection** ([app.py:164-203](app.py#L164-L203)) Once anchored, the system performs **streaming word-level comparison**: ```python # Alignment process using edit distance opcodes: 1. Split canonical verse into tokens: ["ุจูุณู’ู…ู", "ูฑู„ู„ูŽู‘ู‡ู", "ูฑู„ุฑูŽู‘ุญู’ู…ูŽูฐู†ู", "ูฑู„ุฑูŽู‘ุญููŠู…ู"] 2. Split transcription into tokens: ["ุจุณู…", "ุงู„ู„ู‡", "ุงู„ุฑุญู…ู†", "ุงู„ุฑุญูŠู…"] 3. Normalize both for comparison 4. Use SequenceMatcher.get_opcodes() to detect: - 'equal': Correctly recited word โ†’ green highlighting - 'replace': Wrong word โ†’ red highlighting with underline - 'delete': Skipped word โ†’ red highlighting - 'insert': Extra word added โ†’ ignored in canonical display ``` **Visual Feedback**: - โœ… **Correct words**: Green text (``) - โŒ **Wrong words**: Red bold underlined text (``) - โŒ **Completely wrong verse**: Red double-underlined (``) **Streaming Display Constraint** ([app.py:192](app.py#L192)): - During **active recitation**: Only shows correctly matched words (prevents showing unrecited text) - After **pause/completion**: Shows all words including errors for review ### 4. Verse Completion & Auto-Advancement **Intelligent Verse Boundary Detection** ([app.py:332-348](app.py#L332-L348)) The system determines when a verse is complete using two methods: **Method 1: Token Exhaustion** ```python canonical_tokens = "ุจูุณู’ู…ู ูฑู„ู„ูŽู‘ู‡ู ูฑู„ุฑูŽู‘ุญู’ู…ูŽูฐู†ู ูฑู„ุฑูŽู‘ุญููŠู…ู".split() # 4 tokens speech_tokens = transcribed_text.split() is_completed = (last_matched_index >= len(canonical_tokens)) ``` **Method 2: Next-Verse Transition Detection** ```python # If user starts reciting next verse while current verse incomplete: 1. Extract leftover tokens after last match 2. Compare with beginning of next verse (verse N+1) 3. If similarity โ‰ฅ75% โ†’ mark current verse complete, auto-advance ``` **Upon Completion**: 1. Finalizes current verse with ayah number marker `๏ดฟูก๏ดพ` 2. Increments expected verse index 3. Carries over leftover tokens to next verse 4. If leftover tokens don't match next verse โ†’ marks as error ### 5. Continuous Streaming Pipeline **Real-Time Audio Processing** ([app.py:222-415](app.py#L222-L415)) ```python Audio Stream โ†’ Accumulation โ†’ ASR Transcription โ†’ Alignment โ†’ Display โ†“ โ†“ โ†“ โ†“ โ†“ 16kHz WAV 0.5s minimum FastConformer SequenceMatcher HTML buffer (streaming) (live tracking) update ``` **Key Features**: - **Silence Detection**: RMS < 0.005 triggers pause (2 chunks = ~0.4s) - **Streaming ASR**: Runs transcription every 0.5+ seconds on accumulated audio - **Live Preview**: Shows real-time alignment even before pause - **Commit on Pause**: Finalizes verse detection only during valid pauses - **State Persistence**: Maintains verse context, partial recitation, and history across chunks ### 6. Error Classification The system distinguishes three error types: | Error Type | Detection Method | Visual Display | |------------|------------------|----------------| | **Wrong Word** | Token mismatch via `replace` opcode | Red underlined word | | **Skipped Word** | Missing token via `delete` opcode | Red underlined word in canonical position | | **Wrong Verse** | No anchor found or <75% next-verse match | Entire verse in red double-underline | ### Performance Characteristics - **Latency**: ~200-500ms from speech to visual feedback (depending on chunk size) - **Memory**: Streams audio in chunks; clears after each pause - **Accuracy**: Inherits 0.14% WER from base model, enhanced by verse-constrained matching - **Robustness**: Handles pronunciation variations through normalization and fuzzy matching This architecture enables the application to function as an **intelligent Quran tutor** that not only transcribes but actively guides users through sequential memorization with immediate, precise feedback. ## ๐Ÿ“ License MIT License