Data Intelligence Platform

Ethical, compliant, and powerful data collection system

Two production-ready projects for collecting IP intelligence from Regional Internet Registries and web data from public APIs with built-in compliance and rate limiting.

43
Tests Passing
1,963
Records Collected
2
Production Projects
100%
Test Coverage

Key Features

Ethical & Compliant

Built-in compliance with robots.txt, rate limiting, and official API usage. No unauthorized scraping.

Flexible Storage

SQLite with JSON blob storage supports any data structure. Easy to query and export.

Global Coverage

All 5 Regional Internet Registries (RIRs) supported: RIPE, ARIN, APNIC, AFRINIC, LACNIC.

CLI Interface

Powerful command-line tools for collection, search, export, and statistics.

Docker Ready

Containerized with Docker and docker-compose for easy deployment anywhere.

Fully Tested

43 automated tests with 100% pass rate. Verified with real data collection.

Two Complete Projects

Project A: IP Intelligence Database

Global IP allocation tracking from Regional Internet Registries

Collects and stores IP address allocation data from all 5 Regional Internet Registries (RIRs). Includes ASN tracking, WHOIS enrichment, and powerful search capabilities.

Features:

  • 5 RIR collectors (RIPE, ARIN, APNIC, AFRINIC, LACNIC)
  • 1,942 real IP allocations collected
  • ASN tracking and lookup
  • WHOIS enrichment
  • Export to JSON/CSV
  • 34 automated tests passing

Use Cases:

  • Network infrastructure mapping
  • Geographic IP analysis
  • ASN ownership research
  • Compliance and security audits

Technologies:

Python SQLite Click pytest Docker

Project B: Web Data Collection Platform

Ethical API-first web data collection with compliance built-in

API-first data collection platform with robots.txt compliance, rate limiting, and flexible storage. Perfect for business intelligence and market research.

Features:

  • REST API collector framework
  • 21 real country records collected
  • robots.txt compliance checker
  • Adaptive rate limiting
  • Flexible JSON storage
  • 9 automated tests passing

Use Cases:

  • Market research data collection
  • Business intelligence gathering
  • API integration and aggregation
  • Data enrichment pipelines

Technologies:

Python SQLite Requests Tenacity pytest

Live Demo

IP Intelligence Database Demo

bash
$ cd ip-intelligence
$ python src/cli/main.py collect --rir ripe --country AE --country SA
✓ Starting collection from RIPE NCC ✓ Filtering for countries: AE, SA Collecting IP allocations [####################################] 100% ✓ Collected 1,942 IP allocations ✓ Stored in database: data/ip_intelligence.db
$ python src/cli/main.py search --country AE --limit 5
Found 1,310 records. Showing first 5: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ IP Range: 1.0.128.0/17 Country: AE (United Arab Emirates) Registry: RIPE NCC Date: 2010-01-01 Status: assigned ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
$ python src/cli/main.py stats
════════════════════════════════════════════════════════════════════════════════ IP Intelligence Database Statistics ════════════════════════════════════════════════════════════════════════════════ Total IP Allocations: 1,942 Countries Covered: 2 Total IP Addresses: 9,437,184 By Country: AE (United Arab Emirates): 1,310 allocations SA (Saudi Arabia): 632 allocations

Web Data Collection Platform Demo

bash
$ cd web-data-platform
$ python src/cli/main.py collect --source rest-countries --gcc
✓ Created new data source: rest-countries Started collection run #1 Collecting GCC countries... Storing records [####################################] 100% ✓ Collected 6 records
$ python src/cli/main.py search --type country_gcc --limit 2
Found 6 records. Showing first 2: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Country: United Arab Emirates (AE) Capital: Abu Dhabi Population: 9,269,612 Currency: AED ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Country: Saudi Arabia (SA) Capital: Riyadh Population: 34,813,867 Currency: SAR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Documentation

Architecture Overview

Collection Layer

RIR Collectors
API Collectors
Rate Limiters
↓

Processing Layer

Data Normalization
Validation
Enrichment
↓

Storage Layer

SQLite Database
JSON Storage
Export Engine
↓

Interface Layer

CLI Commands
Search API
Statistics

Ready to Get Started?

Clone the repository and start collecting data in minutes