dashboard.opiq.ee

Opiq is an online platform that connects learners and publishers. Public school students use the studying platform in 5+ countries. Publishers can create and distribute content. The platform has more than 500 textbooks digitalised.

With 70% of Estonian students using the platform, Opiq needs a robust data pipeline for site visitor analytics. This data is the basis of revenue share calculations and platform usage analysis, making it business critical. Besides the tens of millions of pageviews from Estonia, the data pipeline needs to process events created in the Finnish, Lithuanian, and Kenyan sites.

As a result, Bitropia built a data processing pipeline that enables 2 strategies.

As the first strategy, for sites with lower volume, a Dockerized container runs a node.js program which consumes Kafka for messages about site usage. These messages are enriched, formatted, parsed, and altered in other ways. Messages are stored in memory until a batch limit is met, which is when they will be directly inserted to AWS Redshift.

For the second strategy, where volume of messages are too large for direct batch insert to Redshift, we save the altered messages in AWS S3 as .json files. AWS Glue is then used to transform the .json files to AWS Redshift at very large scale.

AWS Redshift is prepared for analysts to offer best querying abilities. Tables have a wide range of columns based on which complex queries can be made with ease. The database also has various materialized views.