Challenges faced

The Company already had an existing portal wherein collecting and processing all posts was completely manual while being devoid of automation and other features. Manually collecting posts from various social media based on certain keywords was becoming challenging and time-consuming. Also, scaling was almost impossible. Another problem was that manually processing all the posts from various sources became extremely complicated and prone to errors. The existing portal also did not support role management as far as client, staff, and admin logins were concerned. Along with that, the client was also facing issues in creating clean and informative visualizations using their existing portal.

Proposed Solution

First, we built a system that automatically collects data from various social media based on certain keywords. We also provided a system for uploading data manually from any other offline source. For processing all the data accurately and efficiently, we used a combination of 3rd party APIs and some custom-built APIs. One potentially major issue that we faced while using AWS Lambda services was that it had many limitations on the number of requests it could handle at a time.

We also built a batch system that sent a particular number of requests at a time to solve the issue of limitations in the number of requests that could be handled at a time. As far as role management was concerned, we built a system where each user assigned to a particular role would have a certain access level. For auto-visualization and reporting in a very clean and informative way for their clients, we also built a customized dashboard by collating required data from sources identified by the client.