--- license: agpl-3.0 language: - en metrics: - accuracy tags: - summarization - news - transformer - bart - distilbart - financial-news - text2text-generation - encoder-decoder datasets: - vblagoje/cc_news - Brianferrell787/financial-news-multisource - Sachin21112004/DreamFlow-AI-Data base_model: - sshleifer/distilbart-cnn-12-6 pipeline_tag: summarization library_name: transformers --- # 📰 DistilBART News Summarizer ## The Complete Story: How This Model Was Built, Why It's Special, and How It Works --- ## 🎯 What Is This Model? (A Simple Explanation) Imagine you have a very long news article, and you want someone to read it and tell you the key points in just a few sentences. That's exactly what this model does! **This model takes a long news article and turns it into a short, easy-to-read summary.** Think of it like: - You give it a 5-page news article - It reads through it carefully - It writes back a 3-4 sentence summary that captures all the important information The special thing about this model is that it's: 1. **Very accurate** - It understands news writing style very well 2. **Very fast** - It works quickly even on regular computers (not just expensive AI servers) 3. **Specialized in news** - It was trained specifically on news articles, so it understands how journalists write 4. **Good with financial news** - It knows market terminology, stock names, economic terms --- ## 🔑 Quick Facts AT A GLANCE | Question | Answer | |----------|--------| | **What does it do?** | Turns long news articles into short summaries | | **How big is it?** | 306 million tiny math calculations (called "parameters") | | **How fast is it?** | 24% faster than larger models | | **What language does it speak?** | English | | **Is it free?** | Yes, under AGPL-3.0 open license | | **Who made it?** | Sachin21112004 | | **How many people used it?** | 3,846+ downloads in the last month | --- ## 🤔 Why Did I Build This Model? (The Story Behind It) ### The Problem When I wanted to summarize news articles automatically, I had a few choices: 1. Use a huge model (like GPT-3) - Expensive, slow, overkill 2. Use a small generic model - Not accurate enough, doesn't understand news style 3. Use a model trained on something else - Doesn't understand financial news or journalism ### The Solution I decided to take a pre-trained model called **DistilBART** (which is already good at summarization) and train it more on: - **Real news articles** from around the world - **Financial news** from 35 years of data (1990-2025) - **57 million+ articles** to give it comprehensive coverage This made it specialized for exactly what I needed: **understanding and summarizing news**. ### The Goal Build a model that: - Understands how journalists write (headlines, structure, facts) - Knows financial terminology (stocks, earnings, markets) - Works fast on regular hardware - Produces high-quality summaries that capture the essence of articles --- ## 🧠 Understanding The Model Architecture (For Everyone) ### What Is a Neural Network? (Simple Version) Think of the model like a very complex system of interconnected switches (called "neurons"). When you pass text through it: ``` Text → Lots of math operations → Understanding → Summary ``` Each connection has a "weight" (like a volume dial) that gets adjusted when learning. A 306M parameter model has **306 million of these dial settings** that get tuned during training. ### How Does This Model "Read" Text? The model doesn't read words like humans do. Instead: 1. **It converts words to numbers** - Each word (or piece of a word) gets assigned a unique number 2. **It processes these numbers through many layers** - Each layer extracts more meaning 3. **It generates output word by word** - Starting from nothing, it predicts one word at a time ### The Two-Part Brain: Encoder and Decoder This model has two main parts that work together: ``` ┌────────────────────────────────────────────────────────────────────┐ │ ENCODER (The Reader) │ │ ─────────────────────────────────────────────────────────────────│ │ │ │ INPUT: "Stock markets surged today as tech companies reported │ │ quarterly earnings that beat analyst expectations..." │ │ │ │ JOB: Reads the entire article, understands what it's about, │ │ extracts the key information, builds a mental "summary" │ │ of the article's content. │ │ │ │ LAYERS: 12 layers of reading/understanding │ │ OUTPUT: A compact understanding of the article │ └────────────────────────────────────────────────────────────────────┘ ↓ [Understanding representation] ↓ ┌────────────────────────────────────────────────────────────────────┐ │ DECODER (The Writer) │ │ ─────────────────────────────────────────────────────────────────│ │ │ │ INPUT: Starts with a special "begin" token │ │ │ │ JOB: Generates the summary word by word, using the encoder's │ │ understanding to make sure the summary matches the article│ │ │ │ LAYERS: 6 layers of generation (condensed from 12 for speed) │ │ OUTPUT: "Tech stocks rallied today after companies reported │ │ earnings exceeding expectations, driving the S&P 500 │ │ up 2.3% to a new record high." │ └────────────────────────────────────────────────────────────────────┘ ``` ### Why 12 Layers For Reading But Only 6 For Writing? **Think of it like this:** - Reading is hard - you need to fully understand everything - Writing is easier - once you understand, you just need to express it The "distillation" process trained the decoder to be more efficient while keeping most of its quality. ### What Is "Knowledge Distillation"? (The Secret Sauce) Here's the key insight: The original BART model has 12 encoder layers AND 12 decoder layers. That's 406 million parameters. I used a technique called **knowledge distillation** to create a smaller but still smart decoder: ``` BIG MODEL (12 decoder layers) SMALL MODEL (6 decoder layers) ───────────────────────── ───────────────────────────── Teacher tells student: Student learns to mimic teacher "Here's the full explanation: by keeping only the most 1+2+3+4+5+6+7+8+9+10+11+12=78 essential parts: 1+2+3+4+5+6=21 (21 ≈ 78? No, but close enough while being 2x faster!) ``` The distilled 6-layer decoder retains **95%+ of the quality** while being **50% smaller**. --- ## 📚 Training Data: Everything I Fed The Model ### Why Training Data Matters (An Analogy) Think of training like teaching a student: - A student who reads 100 textbooks → Understands basics - A student who reads 1,000 textbooks → Understands well - A student who reads 57,000,000 articles → Becomes an expert More relevant training data = Better at the task ### Dataset 1: CC-News (708,241 Real News Articles) | Property | Details | |----------|---------| | **What it is** | Real news articles scraped from news websites worldwide | | **Source** | Common Crawl (a massive web archive) using a tool called "news-please" | | **Time period** | January 2017 to December 2019 | | **Quality** | Professionally written, edited journalism | | **Topics covered** | Politics, business, technology, sports, entertainment, world news | **Sample article structure:** ```python { 'title': 'Tech Giants Report Record Quarterly Earnings', 'text': 'Major technology companies reported record earnings...', 'date': '2019-04-15', 'domain': 'www.reuters.com', 'url': 'https://www.reuters.com/...' } ``` **Why this matters:** The model learns how professional journalists write - their style, structure, and how they present facts. ### Dataset 2: Financial News Multi-Source (57.1 Million Articles!) This is the **BIG WIN** for this model. | Property | Details | |----------|---------| | **Size** | 57,100,000 articles | | **Time coverage** | 35 years (1990 to 2025) | | **Sources** | 24 different financial news datasets combined | | **Total data** | 21.4 GB of news content | | **Special feature** | Trading-aware date handling for accurate chronology | **Sources included:** | Source | What it provides | |--------|------------------| | Bloomberg/Reuters | Major financial news from 2006-2013 | | CNBC Headlines | Business TV coverage 2017-2020 | | Yahoo Finance | Market data and articles 2017-2025 | | S&P 500 Headlines | All stock-related headlines 2008-2024 | | DJIA Headlines | Dow Jones Industrial Average news | | Reddit World News | Crowd-sourced news perspectives | | NYT Headlines | New York Times coverage 1990-2020 | | All The News | Comprehensive US news coverage | | And 16 more... | Various financial and general news | **Why this matters:** After training on 57 million financial news articles, the model becomes an expert in: - Stock market terminology - Earnings reports and financial statements - Central bank policy (Federal Reserve, ECB) - Trading strategies and market movements - Financial entity names (tickers, exchanges, regulators) ### Dataset 3: DreamFlow-AI-Data (21 Custom Samples) | Property | Details | |----------|---------| | **Size** | 21 examples | | **Purpose** | Intent alignment for specific use cases | | **What it does** | Helps the model understand user intent | This custom dataset was used for fine-tuning the model to understand different summarization intents. ### The Combined Advantage ``` TRAINING DATA BREAKDOWN ═══════════════════════ ┌─────────────────────────────────────────────────────────┐ │ Financial News Multi-Source │ │ ████████████████████████████████████████████████████ │ │ ████████████████████████████████████████████████████ │ │ ████████████████████████████████████████████████████ │ │ ████████████████████████████████████████████████████ │ │ 98.8% — 57,100,000 articles │ └─────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────┐ │ CC-News │ │ ████████████ │ │ 1.2% — 708,241 articles │ └─────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────┐ │ DreamFlow-AI-Data │ │ ▌ │ │ (special "start" token) WRITING STEP 1: Looking at encoder's understanding + start token → Decides next word should be "Tech" → Generated: "Tech" WRITING STEP 2: Looking at encoder's understanding + "Tech" → Decides next word should be "stocks" → Generated: "Tech stocks" WRITING STEP 3: Looking at encoder's understanding + "Tech stocks" → Decides next word should be "rallied" → Generated: "Tech stocks rallied" WRITING STEP 4: Looking at encoder's understanding + "Tech stocks rallied" → Decides next word should be "today" → Generated: "Tech stocks rallied today" ... (continues until summary is complete) ... WRITING STEP ~50: → Decides next word should be " " (end token) → Generation complete! ``` **The key mechanism - CROSS-ATTENTION:** Every step, the decoder looks back at the encoder's understanding to make sure the summary stays faithful to the original article. ### Step 5: CONSTRAINTS SHAPE THE OUTPUT Several rules make sure the summary is good: | Rule | Value | Why It Matters | |------|-------|----------------| | **max_length** | 150 | Don't make it too long | | **min_length** | 40 | Make sure it's substantive | | **no_repeat_ngram** | 3 | Prevents "the the the the" problems | | **length_penalty** | 2.0 | Encourages helpful length | | **num_beams** | 4 | Quality vs speed balance | | **early_stopping** | true | Stop when done naturally | ### Step 6: NUMBERS BECOME WORDS AGAIN (DECODING) The model's output is still numbers (token IDs). This gets converted back to readable text: ``` [5678, 9012, 3456, 7890, ...] → "Tech stocks rallied today as major companies reported earnings exceeding expectations..." ``` ### THE FULL JOURNEY ``` ┌────────────────────────────────────────────────────────────────────────┐ │ YOUR NEWS ARTICLE │ │ "Global financial markets experienced significant gains..." │ └─────────────────────────────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────────────────────────────────┐ │ STEP 1: TOKENIZATION (Words → Numbers) │ │ "Global" →...
