Big Data management system

Objective:

The customer wanted us to develop big data databases forming an enterprise business data management system providing companies with the ability to use them with familiar business intelligence tools. The system must be reliable and stable. Also, we needed to make a functional and beautiful graphical interface design of the system.

Solution:

The project began with gathering and analysing client’s requirements and making the estimation of the work scope. The customer provided us with a detailed technical specification and was strongly involved in the project during the working process.

The main technical tasks were:

• connect big data sources and SQL database
• create a user-friendly and engaging interface  design
• ensure control of the main system parameters

Once the estimation was made and it was approved, we started the development process. The first stage was aimed onto the design creation, which we made for one month. Then we moved to the next step, the development of big data management system. For frontend part, we chose React as it suits best for building scalable and reliable projects.

Also, to enable effective system functioning we needed to ensure management of multiple servers. For that purpose, we built express-based Node.js server representing a layer between the end users and data management servers.

To make the system stable and reliable, we defined the principal logic elements as discrete files with test coverage, running every time the application was developed or deployed.

One of the challenges we faced were the changes in the requirements appeared during the project, so we had to implement them in the system design and functionality. And thanks to that flexible decision we managed to reduce our expenses on the main system components’ changes.

The third stage was about application testing and took 5 weeks. Besides QA, the system was tested on the end users. After we gathered and processed their feedback and made some enhancements.
The customer was also involved in the testing process, checking it for functionality implementation, bags, and evaluating our work at the end of each iteration.

Result:

We built a stable and quality big data management system and met all customer requirements. The usage of such up-to-date technologies as React, Redux, Typescript, and Node.js allowed us to ensure a high level of application scalability. For us, the project became a useful experience in developing and integrating Big Data software.