Fajri Abdillah

Ebizu App Publisher Pro

Medium Article

AWS Big Data Journey — from zero to hero

Ebizu App Publisher objective is to provide an app analytics, engagement tool that focuses on highly accurate location intelligence, deep insights on users and monetizing mobile audience data. Ebizu own and operate the largest location and beacon network in Southeast Asia that can be utilize to trigger actions or collect data.

The project itself is hosted on AWS. Highest traffic is ~50 Million hit endpoint / Day. And we receive average of 5 GB / Day in S3 with CSV format.


  1. Simple & easy SDK implementation for Android & IOS
  2. Build campaign based on Geofence, Beacon, Dwell Times or Custom Data and schedule its delivery time
  3. Send notification via various channel such as Push Notification, Email and SMS
  4. Advanced Analytics to get better understanding of user behavior
  5. Monetize without changing user experience, even from inactive users


  • Database Design & System Architecture

    The database will be use PostgreSQL and hosted on AWS RDS. Why Postgre? Because we need JSON feature, and postgre has JSONB. At first JSONB was great, but we are not following "Use the right tool for the right things". So we move Cold Data to S3 via Kinesis Firehose. RDS used for Warm Data.

  • Heavy Write

    Read:Write ratio is 2:8. Ebizu App Publisher will capture every data, and send it into Realtime Stream and Batch Stream.

  • Unlimited Background Jobs

    To make the API still response fast, every process is moved into background jobs, fortunately Lumen has this out of the box. We are heavily using AWS SQS, and Redis for long delay Job.

  • Auto Scaling Servers

    We have this issue with background jobs server, or we called cli servers. We use SQS for auto scaling trigger, based on how many on current queue.

  • Build the fastest API

    With default settings of elastic beanstalk we can achieve ~50 Million hit endpoint per day, but we still can improve it. We have plan to use Swoole.

  • Data Warehouse

    Because we have our data in S3 we can utilize Athena to query our data.

  • Send Notification

    AWS SNS to the rescue.

  • Geofence

    PostGIS to the rescue, but we found another awesome tool to replace this job, tile38.


    We can make an event when user enter geofence, for example, send push notification, send sms, send phone call, etc.


2016 Jan - 2017 Aug


  1. Route53
  2. EC2 with Elastic Beanstalk (PHP with Laravel)
  3. Lambda (Python with Zappa)
  4. Yii 2 Framework (Admin Panel on EC2)
  5. RDS PostgreSQL
  6. Kinesis Stream & Firehose
  7. SNS (Push Notification)
  8. Elasticache Redis
  9. S3
  10. SQS
  11. Athena, Redshift & EMR




 49 Million Hit a day

 S3 Data

 S3 Daily Data

 11 Million SQS


Lesson learned

Architecting Big Data Application on top of AWS is awesome.