RSSMonster Tests Small AI Models, Finds Local Can Match Cloud

A developer replaced OpenAI with Alibaba's Qwen and the new ModernBERT on a self-hosted RSS reader, cutting costs from around €10 per month to zero per-request fees. The experiment found small local models can match cloud APIs for specific tasks, but switching models requires reworking the system around them.

Bottom line — The developer concluded small local models are viable when given one narrow job, not as general-purpose replacements.

Go deeper (5)

• The developer runs RSSMonster, an open-source RSS reader, and was spending about €10 per month on OpenAI API calls for embeddings and classification, according to their blog post on DEV Community.
• They tested Alibaba's Qwen for embeddings and summaries, and ModernBERT — a new encoder model from Answer.AI, LightOn, and collaborators — for classifying article tone and promotional content.
• In benchmark tests, neither OpenAI nor Qwen clearly won; both passed regression tests but 'behaved differently,' with Qwen grouping more articles together and finding broader relationships, per the developer's analysis.
• ModernBERT, introduced in a paper presented at ACL 2025, was trained on 2 trillion tokens with a native 8192-token sequence length and claims state-of-the-art results on classification and retrieval tasks, per the paper by Warner et al.
• The developer found switching models required adjusting 'similarity thresholds and fallback behavior' per model, and that trimming input text to its essence improved results more than feeding the model more content.

Read the reporting