diniscruz.ai / writing / Projects and Innovation Lab

Technical Briefing: Web Content Filtering Project

By Dinis Cruz and ChatGPT Deep Research · · 44 min read

PDF LinkedIn post

Contents · 8 sections
  1. Introduction and Project Overview
  2. Key Design Principles
  3. End-to-End Pipeline Workflow
  4. Caching, Performance Optimizations, and Incremental Updates
  5. Technical Components and Tools
  6. Provenance and Explainability Features
  7. Future Extensions and Opportunities
  8. Conclusion

Introduction and Project Overview

This technical briefing outlines the architecture and design of the Web Content Filtering Project, which aims to give users fine-grained control over the content they see as they browse the web. The project’s core idea is to intercept and dynamically modify web pages in real-time, allowing unwanted content to be filtered out and relevant content to be highlighted. By doing so, users can personalize their web experience – for example, hiding negative news or showing only articles about their favorite sports teams – all without requiring changes from the websites themselves.

This capability is aligned with emerging trends in generative AI and semantic web technology. It leverages recent advances in Large Language Models (LLMs) and knowledge graphs to achieve functionality that would previously have required enormous engineering effort. In essence, we use LLMs to interpret and label web content, then represent both the content and the user’s interests as semantic graphs. By bridging these graphs with deterministic algorithms, the system can filter or transform content on the fly in a transparent and explainable way. Crucially, once the initial AI-driven analysis is done, the browsing experience involves no inline LLM calls – meaning after the first run, pages load quickly using cached decisions and do not depend on expensive AI processing for each view.

We will implement several Minimum Viable Products (MVPs) to validate this technology. One MVP will focus on sentiment-based filtering on a news site (e.g. the BBC News homepage): certain headlines or snippets deemed “negative” in tone can be blacked out or removed, giving the user a positivity-filtered news feed. Another MVP will provide topic-based customization on a sports news page: for example, a user interested only in football (soccer) and basketball – and specifically following teams like Wrexham and Benfica – will see those stories normally, while all other sports news is hidden or muted. These scenarios are both practical and personally relevant, demonstrating everyday use-cases for the content filtering technology.

In the following sections, we detail the system’s design principles, the end-to-end processing pipeline, the technical components (including open-source tools and past research contributions being leveraged), and how we ensure performance, determinism, and explainability. Throughout, we will reference prior work by Dinis Cruz on GenAI pipelines, semantic graphs, and content provenance, as this project builds directly on those foundations.

Key Design Principles

Before diving into the implementation, it’s important to understand the guiding principles of our approach:

With these principles in mind, let’s walk through the actual workflow of how a web page goes from the origin server to a filtered result in the user’s browser, detailing each technical step in the pipeline.

End-to-End Pipeline Workflow

The content filtering pipeline can be thought of as a series of transformations on the data (similar to an ETL pipeline, but for web content). We can label the stages as Load, Extract, Transform, and Save (mirroring the LETS methodology). Here’s the step-by-step breakdown:

1. Page Load via Proxy (Intercept and Save Raw Content): When the user navigates to a webpage (say https://www.bbc.com/ for the BBC News homepage), the request is routed through our proxy. The proxy (which runs either locally on the user’s machine or on a server as a cloud service) forwards the request to the real website and fetches the page. When the response (HTML content) comes back, the proxy first saves a copy of the raw HTML to persistent storage (for example, an AWS S3 bucket or a local directory). This stored copy is indexed by URL and timestamp. We effectively create a database of web pages we’ve seen, where each page’s content is stored in a timestamped folder (this allows versioning – we can keep snapshots of how a page changes over time). The idea of storing raw content immediately is borrowed from our content capture architecture, where separating the capture from processing is valuable. As described in a related project document, “the captured content is then sent to a stateless backend and stored in AWS S3… preserving raw web content with minimal server-side processing, enabling future analysis such as semantic knowledge graphs or provenance checks”. By hashing the content and using it as an identifier, we also avoid storing duplicates – if the same content was saved previously, we can just reference the existing record. At this point, we return the HTML content (unmodified) to the pipeline for further processing, but note that we haven’t sent anything to the user’s browser yet – the response is being held and transformed within our system.

2. Parsing HTML to a Typed Structure: Next, the raw HTML is parsed into a structured form that our code can easily manipulate. We utilize the OSBot TypeSafe framework for this. OSBot (an open-source toolkit developed by Dinis Cruz) has utilities to work with web content and define type-safe models. We create a representation of the page’s Document Object Model (DOM) as Python objects – essentially mirroring the HTML hierarchy (elements, attributes, text) in a Python class structure. Each HTML element becomes a node/object with properties (tag name, attributes, parent/children relationships). Text within elements is captured as separate text nodes. By using TypeSafe classes, we ensure that all elements and their relationships are strongly typed and validated, which prevents errors down the line. This step is crucial because it moves us from dealing with raw text (fragile string parsing) to dealing with in-memory objects that can be traversed and analyzed logically. It’s similar to how a browser itself parses HTML, but here we have our own representation we can work with in Python. According to the LETS pipeline principles, we then save this parsed DOM structure (for example as a JSON file or a serialized Python pickle) to our storage. This might be stored as something like page_dom.json. By persisting it, we can skip this parsing step on future runs if the same raw HTML is encountered. It also gives us an artifact to inspect for debugging.

Technical note: OSBot’s TypeSafe system was designed to allow easy conversion between JSON and Python objects, and it leverages Pydantic or similar libraries under the hood for (de)serialization. This means after parsing, we could dump the Python DOM object to a JSON file, and later reload that JSON into the same Python object model, guaranteeing consistency. Using a type-safe model for the DOM also makes it easier to traverse and query (we can search for elements of a certain type, or find text in certain sections, with proper tree relationships).

3. Converting DOM to Graph Representation: With a typed DOM in memory, the next step is to convert this into a graph data structure. We use MGraph-DB (MGraph-AI), an open-source memory-first graph database library, to construct a graph where each node corresponds to a piece of content (e.g., an element or text segment) and edges represent relationships (e.g., “X is child of Y” in the DOM tree). At first, this graph is basically a representation of the HTML structure itself – think of it as the DOM tree turned into a graph of nodes (div, span, p, img, etc.) with parent-child edges. We pay special attention to text nodes: when parsing, we introduced a slight augmentation to the DOM model by creating explicit Text nodes for stretches of text, rather than leaving them as just string values. Each Text node is linked to its container element. This way, every piece of textual content in the page is an addressable node in our graph.

Once the graph is built, we persist it to storage as well (e.g., page_graph.json). MGraph-DB allows us to serialize the graph to JSON easily, since it was designed to treat JSON as the storage format for graphs. In fact, MGraph’s design decision was to keep graphs in memory during computation for speed, but persist every update to the file system as JSON, making the file system the source of truth. This gives us the best of both worlds: the performance of in-memory operations with the reliability and debuggability of persistent storage. Every node and edge can be saved, and we can even diff these JSON files to see changes over time. At this stage, the graph likely contains a few thousand nodes even for a moderately complex page (every paragraph, link, image, etc., plus each text snippet becomes a node). This is the foundation on which we’ll do semantic analysis.

4. Extracting Textual Content: Now that we have the full page graph, we extract the subset of that graph that is relevant for text analysis. Not all nodes are equal from a content perspective – for filtering decisions we mostly care about textual content (the words on the page). So we perform a graph query or traversal to collect all Text nodes (nodes representing actual readable text) and, importantly, gather their surrounding context (such as their parent element or section). The result of this is essentially a list (or subgraph) of textual content pieces. Each item might include, for example, the text string “Manchester United wins FA Cup final” and metadata like: it was inside an <h2> headline element within a <div id="sport-section">. We then structure this extraction as a new “Content Items” graph – where each content item node contains the text and perhaps links (edges) to the section or parent nodes that situate it in the page. We save this content-focused graph (let’s call it the Content Tree for the page) as another JSON (page_content_graph.json). Essentially, we are transforming the raw HTML graph into a distilled form that is easier to feed to an AI and reason about. This transformation is flexible – during development we may tweak what metadata we carry with each text (e.g., we might include the HTML tag hierarchy as part of the content item’s properties, or an XPath/CSS selector to locate it on the page when we need to reconstruct). The key is we’ve isolated all the textual pieces that might be individually classified.

5. LLM Semantic Classification of Content: This is the most computationally intensive step, and where the generative AI power comes in. For each textual content node we extracted, we need to classify or annotate it according to the filtering criteria. In our current MVP scope, we have two parallel classification tasks:

It’s worth noting that this architecture easily generalizes to other classification dimensions. For example, we could also classify each content piece by topic domain (politics, entertainment, technology, etc.), by geographic relevance (mentions of countries or cities), by factuality or source credibility, and so on – depending on future filtering needs. Each would be another property or subgraph attached to the content nodes, derived from LLM analysis. The modularity of having multiple small LLM calls (one per content piece, possibly batched, and specialized per property) follows the principle of using the right tool for each task, rather than one giant prompt that tries to do everything.

All results from this stage are saved. For instance, we’ll have files like page_content_sentiment.json and page_content_topics.json or a combined page_semantic_graph.json that stores the classified graph. Each piece of text content now has machine-understandable annotations: sentiment label, topic category, and associated entities (sports/team names in our use case). This completes the Extract phase (we’ve extracted knowledge from raw text) and sets the stage for the Transform phase, where we actually decide what to show or hide.

6. Building the User’s Persona/Preference Graph: Equally important to analyzing the content is understanding the user’s preferences – i.e. the filter criteria. We capture this in a Persona Graph for the user (or for a given filtering mode). A persona graph is essentially the mirror of the content semantic graph, but for the user’s interests. For example, if the user only cares about Football and Basketball (and within football, specifically the Wrexham and Benfica teams), we will create a graph that has nodes for “Football” and “Basketball” (maybe as sub-nodes of a generic “Sports” interest), and child nodes for “Wrexham” and “Benfica” under football > teams (and perhaps any other specific teams/players the user follows). If the user also indicated they want only positive news, that preference might be encoded as a node or property in their persona graph (e.g., a node labeled “Prefers Positive Content” or simply a rule that negative sentiment = not relevant to this persona). The persona graph can be constructed manually from user input (say, a settings UI where they check boxes of interests) or it can be semi-automated. In future iterations, we could use an LLM to help expand a user’s description into a richer graph – for instance, if a user says “I’m interested in Portuguese football”, an LLM could infer that likely teams of interest might include Benfica, Porto, Sporting, etc., and add those. This is similar to how persona profiles were created in the InsightFlow project: “take a list of interest keywords and use an LLM to expand them into a richer graph of related concepts”. In our first version, we will keep it simpler and directly represent what the user specifies, but we keep the door open for LLM-assisted persona building, which can surface non-obvious interests (e.g., linking “fintech” to related concepts like “blockchain” or, in sports, linking “La Liga fan” to specific clubs in that league the user didn’t explicitly list).

The persona graph is stored similarly in JSON (e.g., user_profile_graph.json). It might have a structure parallel to the content ontology. For instance, it could mirror the sports taxonomy so that matching can be done via common category names or IDs. If the user is in a certain “mode” (like positivity-filter mode), that could also be represented here as a boolean flag or a mode identifier node.

7. Relevance Mapping – Matching Content to Persona: Now comes the decisive step – determining which pieces of the page content are relevant to the user (and thus should be shown) and which are not (to be filtered out). We have all the ingredients: a semantic graph of the page’s content (with labels like sentiment and topics on each text node) and the user’s persona graph (with labels of what they care about). The simplest form of matching is to check for overlaps between these graphs. In practice, this means for each content item, we ask questions like:

We can implement this matching logic in code by traversing the content items and checking their attributes against the persona graph. In many cases, a direct graph algorithm can do this: e.g., graph intersection (find nodes in content graph that have a relationship to any node in persona graph). Since we gave content nodes explicit links to topic entities (like an edge from a content node to the “Football” node in a taxonomy), and the persona graph likely has a “Football” node if the user cares about it, finding a match can be as easy as seeing if there’s a path from the content node into the persona graph. MGraph-DB supports queries and set operations on graphs which we can use to automate this. In some scenarios, we might still use a lightweight LLM prompt to refine the matching – for instance, to evaluate if an article is strongly relevant or just tangential. However, the aim is that this step is largely non-ML, purely data-driven. In the MyFeeds project, a similar step was done with an LLM to ensure flexibility with synonyms and context (the LLM could recognize connections that exact graph matching might miss, like mapping “EU regulation on privacy” content to an interest in “GDPR compliance”). For our immediate use cases, synonyms are less of an issue (sports team names are explicit, sentiment is a direct flag), so we can likely do this deterministically.

Regardless of how the matching is implemented, the outcome is a determination for each content piece: Relevant (keep) or Not Relevant (filter out). We compile these results into a mapping result object – effectively a list of content node IDs that should be shown or hidden. We also record why each was classified that way, by linking back to the matching criteria. For example, we might produce a JSON that says:

{
  "show": [
    {"node_id": 42, "reason": "Matches interest 'Football' (mentions Benfica)"},
    {"node_id": 57, "reason": "Positive sentiment content"}
  ],
  "hide": [
    {"node_id": 13, "reason": "Negative sentiment content"},
    {"node_id": 21, "reason": "Topic not in user's interest (Cricket)"}
  ]
}

All this information is stored (e.g., page_filter_results.json). Storing the mapping results is important for provenance – it forms the basis of our explanation to the user. In fact, by maintaining the connections between content and persona in a graph structure, we build a provenance trail that can justify each inclusion or exclusion. For example, “Article X is shown because it connects to 3 topics you care about: X, Y, and Z” (straight from the InsightFlow methodology). In our case, it might be simpler (one topic or one reason), but the concept is the same. This explicit mapping is something we can present to users for transparency.

8. Page Reconstruction (Applying the Filter): Now we have the original page content and a list of what should be visible or hidden. The final step is to reconstruct the HTML to reflect the filtering decisions. Since we still have the original DOM/graph in memory (or we can reload it from the saved state), we can go through it and for each content node that was marked “hide”, we remove or mask that element. There are a couple of ways to do this. A straightforward approach is: for a text node to hide, we can replace its text with a placeholder (e.g., “████” or an empty string) in the HTML. Alternatively, we could remove the entire HTML element containing that text (which might be better if we want to collapse space). Initially, we might choose a conservative route like replacing text with a black bar or a note like “[removed]”, so the user can see that something was there but is being hidden. For visible items, we may also choose to highlight them (for example, outline the preferred sports news in green). Highlighting is not strictly necessary, but it can be a nice visual confirmation in demo mode that “these are the items of interest”. All such modifications are done in the DOM structure and then serialized back to HTML markup.

During this reconstruction, we can also inject the top banner we discussed. The banner can be a simple <div> at the top of the body that says something like: “WebContentFilter active: X items removed, Y items highlighted. [View Details]”. The “View Details” link could point to a local page (perhaps served by the proxy or a static file) that reads the page_filter_results.json and displays the provenance information in a friendly way (e.g., a list: Removed Headline “XYZ” – Reason: Not in Sports interests). This closes the loop on transparency: the user not only experiences the filtered content but can also inspect exactly what was done.

After injecting the banner and finalizing the HTML, the proxy sends this modified HTML to the user’s browser. From the user’s perspective, the page loads normally except they notice some content missing and the banner present. The filtering is complete!

To summarize the pipeline in a simple flow: Browser Request → Proxy fetches page → Save raw HTML → Parse to DOM → DOM to Graph → Extract Text Nodes → LLM classification (sentiment/topics) → Save semantic graph → Load Persona graph → Match graphs for relevant content → Modify HTML (remove/hide content) → Deliver to Browser. Each of those arrows represents data saved and available for debugging or reuse.

It’s worth highlighting how this approach scales and remains maintainable. Each step is separate and outputs files (HTML, JSON graphs, etc.). If something goes wrong in the final output, we can trace back: check the mapping results, check the semantic classifications, check the content extraction, etc. This modular design is inspired by earlier pipeline work. For example, in MyFeeds.ai, the processing was split into numerous small steps (fetch feeds, extract text, LLM to entities, build graph, compare to profile, etc.), each writing out its result. This made the system highly debuggable and resilient: “if something failed at step 5 for article X, the engineer could retrieve article X’s JSON from step 4 and investigate... since each step is idempotent on its input file, the system can retry or resume failed steps without starting over”. We are employing the same strategy here.

Caching, Performance Optimizations, and Incremental Updates

After the first run of a given page, subsequent visits should be much faster, as the heavy lifting is already done:

In summary, the system uses a combination of caching, content hashing, and splitting work into independent pieces to ensure that after the initial investment of processing a page, subsequent operations are extremely fast. By treating intermediate results as reusable assets (much like a compiler caches object files), we minimize redundant work. This approach was proven in the MyFeeds.ai pipeline, where each transformation’s output was saved and could be reused or inspected, leading to determinism and efficiency. In that project, the result was a personalized content feed that was complex but behaved deterministically, with every intermediate reasoning step materialized for debugging – exactly what we strive for here in the context of web browsing.

Technical Components and Tools

This project stands on the shoulders of open-source tools and past research, particularly those developed by Dinis Cruz and collaborators. Here we credit and describe the key components being utilized:

In summary, our tech stack is predominantly Python-based, with heavy use of JSON as a lingua franca between stages, and built on open standards and open-source projects (ensuring we can share parts of this work with the community or integrate improvements from others). By using and crediting these tools (OSBot, MGraph-DB, etc.), we also plan to contribute back by highlighting their capabilities in a new domain (web filtering) and potentially raising issues or extending them as needed for our purposes.

Provenance and Explainability Features

As mentioned earlier, one of the standout features of this project is its ability to explain its own behavior. Here we detail how provenance data is captured and how it might be surfaced to users or developers:

The UI could allow the user to toggle a switch next to each reason to say “always/never filter this kind of content”. That could provide feedback to the system (for example, if the user toggles “never filter negative news”, we’d remove the sentiment rule from their persona).

Another aspect of explainability is to show the strength of relevance. In some systems like InsightFlow, they computed a relevance score (e.g., 8.5/10) and could cite multiple matches (article X is 8.5 relevant because it hits 2 big topics you care about). We can incorporate a simple scoring (e.g., +1 for each interest matched, -1 for each filter criterion failed) to give a sense of confidence. But for transparency, listing matches is probably sufficient. The graphs we store essentially contain all the raw info needed to compute such a score or explanation.

Future Extensions and Opportunities

While the core architecture is now in place for sentiment and sports-interest filtering, there are many directions this project can grow, both technically and in terms of features:

In conclusion, the Web Content Filtering Project is not just about hiding a few elements on a page – it’s a framework for dynamic, personalized web experiences. By leveraging GenAI to understand content and using graphs to make decisions explicit and traceable, we set a foundation that can be extended in myriad ways. Our current focus is delivering the promised MVP (sentiment and sports filtering) reliably and clearly. But we’re also keeping an eye on the broader vision: a future where users have complete control and insight into the information they consume, powered by open-source tools, transparent algorithms, and AI assistance where it adds value.

Conclusion

The Web Content Filtering Project showcases a novel integration of Generative AI, knowledge graphs, and web technology to empower users in tailoring their online content consumption. By intercepting web pages and transforming them in real-time, we enable features like hiding unwanted content and spotlighting what matters most to an individual user – all done in a way that is explainable, deterministic, and efficient.

This technical briefing has walked through the detailed architecture: from the initial page capture and parsing, through multi-stage AI-driven analysis (sentiment detection, topic classification) using LLMs with structured outputs, to the construction of semantic graphs that allow precise matching against a user’s interest graph. We’ve emphasized how the system saves each intermediate result (following a LETS pipeline approach) to achieve transparency and reliability. This approach, influenced by Dinis Cruz’s prior research on provenance in AI systems, means that every filtered page comes not as an inscrutable magic trick, but as the end result of a series of traceable transformations – any of which can be inspected or audited. The use of open-source tools like OSBot and MGraph-DB is not only a practical choice but also grounds the project in a wider community of graph and AI innovation. We have built on these tools and in doing so credited the past work that made them available, from OSBot’s type-safe classes for LLM JSON handling to MGraph’s memory-first, JSON-backed graph model.

The initial implementation will provide immediate user-facing value in the form of customizable news filtering, but the architecture is flexible and extensible. As we move forward, we foresee expanding the system to cover more use cases, refining the ontologies and taxonomies that drive classification, and possibly collaborating with content publishers for even richer data integration. We will also gather feedback from users and the technical team to iterate on the design – for example, fine-tuning the balance between LLM usage and direct graph logic, or improving the UI/UX of the in-browser modifications and explanations.

In summary, this project represents a step toward a more personalized and user-centric web, where the user is in control of content filtering criteria (rather than relying on each website’s one-size-fits-all design), and where AI serves as a powerful assistant to implement the user’s intent, but does so under the user’s guidance and with full accountability. By coupling GenAI with deterministic graphs and by treating data as a first-class asset (storing everything for reuse and inspection), we deliver a system that is both cutting-edge and trustworthy.

We look forward to building this in collaboration with the team and, as we do so, continuing to document and share the lessons learned. The outcome will not just be a useful product but also a reference architecture for GenAI-driven content personalization that others can learn from or replicate, further crediting and building upon the open-source and research contributions (like those by Dinis Cruz) that have paved the way.

Sources Cited:

Released under CC BY 4.0. First published on docs.diniscruz.ai; this page as markdown.

← Follow-Up Technical Vision: Optimizations, Deployment, and SecurityUser-Driven Semantic Persona Graphs Powered by GenAI →