Big Data Archives - IBA Group - USA https://us.ibagroupit.com/tag/big-data/ Mon, 14 Oct 2024 11:40:56 +0000 en-EN hourly 1 https://wordpress.org/?v=6.5.5 AWS cloud migration for a global company https://us.ibagroupit.com/cases/aws-cloud-migration/ Mon, 24 Oct 2022 16:01:25 +0000 https://us.ibagroupit.com/?post_type=cases&p=9111 The post AWS cloud migration for a global company appeared first on IBA Group - USA.

]]>

Background

А huge global company with a strong technology focus turned to IBA Group with the task of transferring their client’s infrastructure to cloud services. At the time of the beginning of cooperation, the end customer of our client used a lot of data storage for the following internal service tasks:

  • production cost tracking;
  • dialer’s information;
  • providing information to dealers,
  • information about clients, employees.

The main part of the infrastructure was based on the Oracle database and the Informatica Power Center ETL tool, as well as many other tools such as Cron, Shell Script, Java, JS.

Several years ago, the company’s management decided to migrate the entire infrastructure to the cloud. During modernization, the company brought production processes to a single standard, increasing its structure, including a unified approach to development, implementing an approved technology stack, and ensuring quality control of development.

Business Challenges

During the modernization process, the end client encountered the following difficulties:

  1. The existing services of the company were slow, which impacted all processes — from data analysis to hiring employees.
  2. Frequent failures due to errors within the services led to a partial loss of data and the inability to make management decisions, prepare accounting reports, prepare data on the movement of warehouse resources, and more.
  3. Permanent delays in providing new services due to the slowdown in development or the impossibility of their implementation, led to a decrease in the company’s attractiveness among customers.
  4. Scaling within the framework of the old infrastructure could not be carried out without extra payments, due to the low flexibility of vendor licensing offers.
  5. Recruitment of the necessary employees such as full-stack developers presented difficulties due to the huge variety of technologies used; cheaper resources were not of the right quality or up to the high standards of the company.
Key Objective #1

Migrate company services to new infrastructure in the cloud.

Key Objective #2

Tailor AWS vendor migration offerings to company services.

Solution Highlights

The customer formed a separate team that was responsible for the innovation and unification of disparate systems into a single infrastructure. IBA Group provided engineers and architects to form custom teams to participate in the adaptation of infrastructure modernization solutions, developing the top-level structure of the entire ecosystem, as well as general approaches and rules for each system separately.

After giving recommendations for the development and auditing of the process, the IBA Group team directly implemented this migration. To improve the quality of decisions made at the architecture design stage, an AWS vendor was involved.

Due to the business’s specific features, each area had its own architecture, for which separate teams were created that were responsible for its adaption and improvement. They were also involved in the transfer of existing services for their area to cloud technologies — AWS Glue, PySpark, ECS, Kubernetes, EMR + PySpark.

One of the important conditions was a “seamless” transition — teams during the migration process ensured that access to existing services remained uninterrupted.

Project Results

Benefits the client received by moving data management to a cloud service:

  1. Increased fault tolerance by fixing bugs and using ready-made cloud-platform services for interaction between different customer services.
  2. Continuity of current services during the migration process was maintained.
  3. Cost reduction in the form of refusal of some server capacities due to the transition to the cloud service.
  4. Flexible planning of financial expenses for scaling due to the pay-as-you-go system provided by the cloud solution.
  5. Reduced time for recruiting specialists in the IT department by unifying the requirements for them, defining clear rules, and specifying the technology stack.
  6. Reduction of the number of narrowly focused specialists without full-time employment due to their retraining in the technology stack adopted by the company.

The post AWS cloud migration for a global company appeared first on IBA Group - USA.

]]>
Development of a Cognitive Platform for Enterprise Data Management https://us.ibagroupit.com/cases/cognitive-enterprise-data-platform/ Mon, 10 Oct 2022 08:43:04 +0000 https://us.ibagroupit.com/?post_type=cases&p=9080 The post Development of a Cognitive Platform for Enterprise Data Management appeared first on IBA Group - USA.

]]>

Business Challenge

Build a secure and scalable data aggregation and analytics cognitive platform that transforms key business processes.

Solution

IBA Group developers worked on the Cognitive Enterprise Data Platform (CEDP). This platform makes it easy to get, share, and integrate enterprise data. Various business units of the company interact with the platform as producers and consumers of data.

Result

CEDP currently contains over 100 enterprise data assets. Datasets include customer master data, product offering master data, unified taxonomy, and more.

CEDP has big data analytics capabilities such as BigSQL, HDFS, DashDB for performance-based analytics, unstructured data annotation capabilities, machine learning, deep learning, and access to IBM Cloud Services.

The post Development of a Cognitive Platform for Enterprise Data Management appeared first on IBA Group - USA.

]]>
Rice Field Analysis System using Drone Photos https://us.ibagroupit.com/cases/big-data-ai-manufacturer/ Mon, 25 Jan 2021 14:36:08 +0000 https://us.ibagroupit.com/?post_type=cases&p=7133 The post Rice Field Analysis System using Drone Photos appeared first on IBA Group - USA.

]]>

BUSINESS CHALLENGE

Develop an application to analyze photos made by a drone while flying over rice fields. Count the number of plants that have sprouted on the plots in the photo.

SOLUTION

The IBA Group team implemented image processing algorithms to accurately calculate the number of objects located on them. For end users, we created an application that processes photos, determines the number of plants on them and generates reports in *.csv and *.pdf formats.

Application algorithm:

  • Create a polygonal image of the region of interest.
  • Download satellite image for it (requires Internet connection).
  • Take a calibration photo (taken at the desired measurement height containing two highly visible objects placed on the ground at a distance of 1 m from each other).
  • Take a photo above the field for measurements.
  • Perform calibration procedure using the calibration image.
  • Generate report.

OUTCOME

Thanks to the application, the customer can receive reliable information about the state of rice fields using drones. The project is continuously improving.

The post Rice Field Analysis System using Drone Photos appeared first on IBA Group - USA.

]]>