Opiq Dashboard

About Opiq

Opiq is an online platform that connects learners and publishers. Public school students use the studying platform in 5+ countries. Publishers can create and distribute content. The platform has more than 500 textbooks digitalised. Every day hundreds of thousands of teachers and students use Opiq in classrooms and at home to learn new subjects, practice exercises, or do tests.

The Challenge

With 70% of Estonian students using the platform, Opiq needs a robust data pipeline for site visitor analytics. This data is the basis of revenue share calculations and platform usage analysis, making it business critical. Besides the tens of millions of pageviews from Estonia, the data pipeline needs to process events created in the Finnish, Lithuanian, Kenyan, and Kazakhstan sites.

The Result

As a result, Bitropia built a data processing pipeline that enables 2 strategies.

As the first strategy, for sites with lower volume, a Dockerized container runs a Node.js program which consumes Kafka messages about site usage. These messages are enriched, formatted, parsed, and altered. Messages are stored in memory until a batch limit is met, which is when they will be directly inserted into AWS Redshift.

For the second strategy, where volume of messages is too large for direct batch insert to Redshift, we save the altered messages in AWS S3 as JSON files. AWS Glue is then used to transform the JSON files to AWS Redshift at very large scale.

AWS Redshift is prepared for analysts to offer best querying abilities. Tables have a wide range of columns based on which complex queries can be made with ease. The database also has various materialized views.

In the frontend, a React.js application serves analytics for key stakeholders: business owners, school administrators, publisher representatives, and governmental officials.