Business Need
Our team created a simple way to forecast based on the financial market data and actual news sources using Machine Learning algorithms. We developed a successful product that allows a basic method of ML algorithm solutions and uses UI to enable control for each step of the process.
Solution
The product consists of several separate services with specific features such as:
–ML Template: base implementations of Spark ML algorithms stored in Git (Scala, Spark, SBT)
–Executor Service: service for downloading and compiling algorithm sources, model training, running on the Spark cluster, saving results to PostgreSQL database (Java, Spring, Hibernate, Gradle)
–Data Service: service for downloading news information (GDELT data) with related stock market data and saving them to Mongo database (Java, Spring, Hibernate, Gradle)
–Front End: user interface for managing algorithm execution through services listed above; providing the ability for results checking (ReactJS, Bootstrap)
–Data Storage: PostgreSQL database; stock market data and news parameters are stored in Mongo database (PostgreSQL, MongoDB)
Every instance of services, databases, and Spark nodes has its own Docker image running under the Rancher container management platform. Spark cluster is based on Mesos and logs are stored in the ElasticSearch instance and visualized by Kibana. CI is managed using Jenkins.
Benefits and Results
Akvelon successfully developed technical architecture and implemented and delivered the solution.
Benefits of our solution:
•A platform with a simple process of ML algorithm submissions
•UI that allows controlling each step from algorithm submission to execution results
•Deployed infrastructure for the developed solution
Technology Used
Scala, Apache Spark, Spark ML, SBT, Java, Spring Framework, Hibernate, Gradle, PostgreSQL, MongoDB, Docker, Apache Mesos, Rancher, Jenkins, ElasticSearch, Kibana, ReactJS, Bootstrap
