Big Data Collecting And Reporting System

Decision making in corporations requires reliable sources of information to identify industry/company trends. The developed solution can provide with insights to available data needed for:

• Investment research and management,
• Competitive intelligence,
• Sales lead generation,
• KPI and performance tracking.

This huge system — the result of joint work of analysts and programmers experienced in finance and big data — collects data with proprietary methods from 300+ named open sources of different specialization, unifies it and prepares for accurate and clear reporting.

The system has a nontrivial volume structure in order to be stable, agile and scalable. By March 2018, more than 360 scrapers (at least one scraper for one data source) operate in parallel to ensure real time data collection and reporting. It has traffic of 20+ terabytes per month and the databases with 8 TB of data overall, while one of database’s tables contains 720 millions of records and weights 450 GB.

As a result, the user gets access to a handy interface with requested statistics, graphs and sortable tables with all controls and filters needed to understand and be empowered by big data.