North Cloud / Open-source infrastructure
From scattered sources to structured content.
Active development
A pipeline for crawling, classifying, searching and distributing articles.
Go / Elasticsearch / Redis / Vue
- 01Collect
- 02Classify & search
- 03Distribute
01 / The problem
The problem
A useful content feed needs more than a crawler. Sources must be managed, articles normalized and classified, and results routed to the consumers that need them.
02 / Constraints
Constraints
Collection, classification and distribution evolve at different speeds. Operators need visibility into each stage, and consumers need a stable way to receive content.
03 / My contribution
My contribution
I develop the service architecture, crawl and classification workflows, publishing routes and management tools.
04 / Decisions & tradeoffs
Decisions & tradeoffs
Separate Go services handle source management, crawling, classification and publishing. Elasticsearch holds searchable content; Redis Pub/Sub distributes routed articles. A Vue dashboard and pipeline events expose operational state.
05 / What is demonstrated
What is demonstrated
The public repository documents the crawler-to-publisher pipeline, rule and ML classification, search, consumer integration and Docker development environment.
06 / Limitations & current work
Limitations & current work
The repository demonstrates the architecture and implementation. It does not establish customer adoption, uptime or a quantified publishing outcome.
Explore the project
Public README and pipeline documentation reviewed September 30, 2026.