Local video transcription and summarisation with Whisper and a quantised LLM

Problem

Watching an hour-long lecture, talk or interview to find out whether it is worth it takes, precisely, an hour. Services that summarise videos send the audio to third-party APIs, with a per-use cost and the question of what happens to the content when it is private: an internal meeting, a recorded class or an interview with personal data.

The goal of this project is an app that runs the whole process locally, on your own GPU: take a video and return a short summary in Spanish together with the full transcript.

My contribution

Group project. My part was:

• Models configurable from the UI. I organised the backend so that the Whisper model, the LLM and their configuration arrive from the frontend with each request. I optimised GPU usage by controlling how models are loaded when the configuration changes from one request to the next.
• Controlling the LLM output. I added control over the output tokens, formatting the result into the expected shape.
• Settings panel in the frontend to choose the models and generation parameters, plus error handling with tracebacks in the backend.

Technical approach

1. 01Video or audioDropped into the chat or recorded in the browser
2. 02FFmpegExtracts the audio: mono MP3 at 16 kHz
3. 03WhisperLocal transcriptionbase · small · medium
4. 044-bit LLMSummarises in Spanish in five sentences at mostQwen 1.5‑14B · Phi‑3 mini
5. 05ResponseFull transcript

A 14B model on a consumer GPU. Qwen 1.5-14B-Chat takes about 28 GB in half precision (FP16). Quantised to 4 bits with NF4 (bitsandbytes), the weights drop to about 7 GB while computation still runs in FP16:

# backend/main.py — loading the LLM (excerpt)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # 4-bit NormalFloat
    bnb_4bit_compute_dtype=torch.float16,  # compute in FP16
)
model = AutoModelForCausalLM.from_pretrained(
    model_name, quantization_config=bnb_config, device_map="auto"
)
# Stop when the assistant's turn ends (ChatML format)
terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|im_end|>"),
    tokenizer.convert_tokens_to_ids("<|endoftext|>"),
]

Instructions to the model. The prompt follows Qwen’s ChatML format and sets explicit rules: summarise only in Spanish, without mixing languages, in no more than five sentences.

Results

• A complete, working app: upload or record, transcription, summary and an expandable transcript in a chat-style interface with light and dark modes.
• All processing happens locally; no audio or text leaves the machine.
• Models and generation parameters can be swapped from the UI without restarting the server.

Evidence

• Repository with the backend (FastAPI) and frontend (React, TypeScript, Tailwind CSS), plus instructions to run it locally.

Professional application

• Summaries of meetings, classes or interviews when content cannot leave the organisation, for privacy or GDPR reasons.
• Deploying quantised open-source LLMs on your own hardware instead of relying on paid APIs.
• End-to-end AI prototypes (model, API and UI) to validate an idea with real users.