Filtering noise from RAG chunks before they poison retrieval

After splitting a document, not every chunk is worth keeping. Ads, nav bars, disclaimers, and duplicates are noise that pollutes retrieval. This walks rule-based, structural, semantic, and LLM filtering strategies, with a Java pipeline you can adapt.

· 5 min read RAG

In a RAG system, after a document is split, not all content is worth keeping — a lot of it is noise (ads, navigation bars, disclaimers, duplicate content) that pollutes retrieval results.

1. The core idea of filtering

Goal: before or after splitting, filter out content that doesn’t help answer the user’s question.

raw document → content filtering → useful content → split → vectorize → retrieve
                                    ↓
                                 noise (discarded)

Filtering has three main dimensions:

  1. Structural noise: web navigation, headers/footers, copyright notices.
  2. Content noise: ads, recommendation links, irrelevant image descriptions.
  3. Quality noise: overly short/long meaningless paragraphs, duplicate content.

2. Common filtering strategies

Strategy 1: Rule-based filtering (simplest, fastest)

public class RuleBasedFilter {
    // 1. Length filter: chunks that are too short (<20 chars) or too long (>2000 chars) are usually low quality
    public boolean isValidByLength(String chunk) {
        if (chunk.length() < 20) return false;   // meaningless short sentence
        if (chunk.length() > 2000) return false; // likely contains too much noise
        return true;
    }

    // 2. Keyword blacklist: content containing these words is likely noise
    private static final List<String> NOISE_PATTERNS = Arrays.asList(
        "click here", "subscribe for free", "all rights reserved", "privacy policy",
        "terms of use", "ad", "sponsor", "related recommendation",
        "cookie", "GDPR", "All rights reserved"
    );

    public boolean isNotNoiseByKeywords(String chunk) {
        String lower = chunk.toLowerCase();
        for (String pattern : NOISE_PATTERNS) {
            if (lower.contains(pattern.toLowerCase())) {
                return false; // hit a noise word, filter out
            }
        }
        return true;
    }

    // 3. Punctuation-density filter: content that's all punctuation/numbers/symbols isn't meaningful
    public boolean hasSubstance(String chunk) {
        int chineseCharCount = 0;
        for (char c : chunk.toCharArray()) {
            if (Character.UnicodeScript.of(c) == Character.UnicodeScript.HAN) {
                chineseCharCount++;
            }
        }
        // content with <20% Chinese chars may not be valid text
        double ratio = (double) chineseCharCount / chunk.length();
        return ratio > 0.2;
    }

    // 4. Deduplication: adjacent duplicate chunks
    public List<String> deduplicate(List<String> chunks) {
        List<String> result = new ArrayList<>();
        String prev = null;
        for (String chunk : chunks) {
            if (prev == null || !isDuplicate(prev, chunk)) {
                result.add(chunk);
                prev = chunk;
            }
        }
        return result;
    }

    private boolean isDuplicate(String a, String b) {
        // simple dedup: similarity > 90% counts as duplicate
        int minLen = Math.min(a.length(), b.length());
        int sameCount = 0;
        for (int i = 0; i < minLen; i++) {
            if (a.charAt(i) == b.charAt(i)) sameCount++;
        }
        double similarity = (double) sameCount / minLen;
        return similarity > 0.9;
    }
}

Strategy 2: Document-structure filtering (for web/PDF)

public class StructuralFilter {
    // For HTML: extract the main body, drop nav/sidebar/footer
    public String extractMainContent(String html) {
        Document doc = Jsoup.parse(html);
        doc.select("nav, header, footer, aside, .sidebar, .ad, .recommendation").remove();
        doc.select("script, style, noscript").remove();
        Element main = doc.selectFirst("main, article, .content, body");
        return main != null ? main.text() : doc.body().text();
    }

    // For PDF: identify and skip headers/footers
    public boolean isHeaderOrFooter(String text, int pageNum, int totalPages) {
        boolean hasPageNumber = text.matches(".*\\d+.*") && text.length() < 50;
        boolean hasDate = text.matches(".*\\d{4}[./-]\\d{2}[./-]\\d{2}.*");
        if (pageNum == 1 && text.length() < 100) return false; // possibly a title
        if (pageNum > 1 && hasPageNumber && text.length() < 100) return true;
        if ((pageNum == 1 || pageNum == totalPages) && text.contains("Copyright")) return true;
        return false;
    }
}

Strategy 3: Semantic filtering (smartest, needs embedding)

Idea: use an embedding model to judge how similar a text chunk is to “useful content.”

public class SemanticFilter {
    private EmbeddingProvider embeddingProvider;
    private static final List<String> POSITIVE_EXAMPLES = Arrays.asList(
        "Company annual leave policy is 15 days per year",
        "Server spec: 16GB RAM, 4-core CPU",
        "Customer complaint process: step 1, log the complaint"
    );
    private static final List<String> NEGATIVE_EXAMPLES = Arrays.asList(
        "Click here for more information",
        "© 2024 Company. All rights reserved",
        "Related recommendation: you may also like"
    );
    private float[] positiveCenter;
    private float[] negativeCenter;

    public SemanticFilter(EmbeddingProvider provider) {
        this.embeddingProvider = provider;
        this.positiveCenter = computeCenter(POSITIVE_EXAMPLES);
        this.negativeCenter = computeCenter(NEGATIVE_EXAMPLES);
    }

    private float[] computeCenter(List<String> texts) {
        float[] sum = null;
        for (String text : texts) {
            float[] vec = embeddingProvider.getEmbedding(text);
            if (sum == null) sum = new float[vec.length];
            for (int i = 0; i < vec.length; i++) sum[i] += vec[i];
        }
        for (int i = 0; i < sum.length; i++) sum[i] /= texts.size();
        return sum;
    }

    public boolean isUseful(String chunk) {
        float[] chunkVec = embeddingProvider.getEmbedding(chunk);
        double simToPositive = cosineSimilarity(chunkVec, positiveCenter);
        double simToNegative = cosineSimilarity(chunkVec, negativeCenter);
        return simToPositive > 0.6 && simToNegative < 0.4;
    }
}

Strategy 4: LLM filtering (most accurate, but costly)

Directly let the model judge whether a chunk is useful:

public class LLMFilter {
    public boolean isUseful(String chunk, LLMClient llm) {
        String prompt = String.format("""
            Decide whether the following text fragment contains valuable information
            (knowledge, facts, explanations, procedures, etc.).
            Ignore: ads, navigation menus, copyright notices, recommendation links, duplicates.
            Answer only "yes" or "no".

            Text: %s
            """, chunk);
        String response = llm.chat(prompt);
        return response.trim().equalsIgnoreCase("yes");
    }
}

3. A complete filtering pipeline

public class DocumentFilterPipeline {
    private RuleBasedFilter ruleFilter = new RuleBasedFilter();
    private StructuralFilter structuralFilter = new StructuralFilter();
    private SemanticFilter semanticFilter;  // optional
    private LLMFilter llmFilter;            // optional

    public List<String> filterAndChunk(String rawDocument) {
        // 1. structural filtering (for HTML/PDF)
        String mainContent = structuralFilter.extractMainContent(rawDocument);
        // 2. split
        List<String> chunks = chunker.chunk(mainContent);
        // 3. rule filtering (quickly drop obvious noise)
        List<String> filtered = new ArrayList<>();
        for (String chunk : chunks) {
            if (ruleFilter.isValidByLength(chunk) &&
                ruleFilter.isNotNoiseByKeywords(chunk) &&
                ruleFilter.hasSubstance(chunk)) {
                filtered.add(chunk);
            }
        }
        // 4. deduplicate
        filtered = ruleFilter.deduplicate(filtered);
        // 5. semantic filtering (optional, costly)
        // if (semanticFilter != null) { ... }
        return filtered;
    }
}

4. Filtering recommendations by scenario

ScenarioRecommended strategyWhy
Internal docs (Word/PDF)rule + structurelittle noise, mainly headers/footers, TOC
Web-scraped contentstructure (extract body) + keyword blacklistweb has the most noise (ads, nav)
Chat logslength filter + dedupmany short messages, lots of duplicates
Academic paperstitle/abstract/conclusion first + body filterneed to recognize paper structure
Pursuing highest qualitysemantic or LLM filteringbest effect, highest cost

5. Java utilities to get started quickly

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.17.2</version>  <!-- HTML parsing -->
</dependency>
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-lang3</artifactId>
    <version>3.14.0</version>  <!-- string utils -->
</dependency>
public class TextQualityChecker {
    public static boolean looksLikeNormalText(String text) {
        int validChars = 0;
        for (char c : text.toCharArray()) {
            if (Character.isLetterOrDigit(c) ||
                "，。！？；：\"\"''（）【】《》、".indexOf(c) != -1) {
                validChars++;
            }
        }
        double validRatio = (double) validChars / text.length();
        if (validRatio < 0.6) return false; // too many special chars
        if (text.matches(".*(.)\\1{5,}.*")) return false; // repeated chars like "......"
        int upperCount = 0;
        for (char c : text.toCharArray()) {
            if (Character.isUpperCase(c)) upperCount++;
        }
        if (upperCount > text.length() * 0.5) return false; // all-caps is noise
        return true;
    }
}

Takeaway

The core of filtering is layered progression:

fastest, lowest cost → rule filtering (length, keywords, punctuation)
   ↓ not enough
  structure filtering (HTML body extraction, PDF header/footer)
   ↓ still not enough
  semantic filtering (embedding similarity)
   ↓ pursue the extreme
  LLM filtering (let the model judge)

Production recommendation: first use rule + structure filtering to drop ~80% of obvious noise, then use semantic filtering for edge cases, and reserve LLM filtering for small-scale, high-quality scenarios (cost is too high otherwise).

Qimiaogan This work is licensed under CC BY-NC-SA. Please credit the source and author when reposting.

Related

Browse all →

Contents