ETL Archives - IBA Group - USA https://us.ibagroupit.com/tag/etl/ Tue, 06 Aug 2024 12:21:50 +0000 en-EN hourly 1 https://wordpress.org/?v=6.5.5 Penetration Testing of Enterprise HR Management System https://us.ibagroupit.com/cases/penetration-testing-enterprise-hr-management-system/ Thu, 01 Jun 2023 07:19:15 +0000 https://us.ibagroupit.com/?post_type=cases&p=9910 The post Penetration Testing of Enterprise HR Management System appeared first on IBA Group - USA.

]]>

Challenges

Customer is one of the largest EU-based IT service providers.

Challenge: identify and eliminate vulnerabilities that could be the cause for unauthorized access to employee data before the production release.

Solution highlights

More than 30 vulnerabilities were successfully eliminated from the software to ensure the secure processing of user data.

Key Results

Based on the high value of the project results, Penetration Testing has been successfully integrated into the Customer’s HR Management System development cycle.

The post Penetration Testing of Enterprise HR Management System appeared first on IBA Group - USA.

]]>
Penetration Testing of AI writer platform https://us.ibagroupit.com/cases/penetration-testing-ai-writer-platform/ Thu, 01 Jun 2023 07:10:27 +0000 https://us.ibagroupit.com/?post_type=cases&p=9909 The post Penetration Testing of AI writer platform appeared first on IBA Group - USA.

]]>

Challenges

Customer is a US-based AI writer service provider with significant number of reputation sensitive customers.

Ensure that end customer’s data cannot be compromised with the delivered solution.

Solution highlights

Around 20 vulnerabilities were successfully eliminated from the software before the product was introduced to the market (only 7 out of ~20 had low threat levels).

Key Results

High risk of personal data loss was reduced to a low level.

The post Penetration Testing of AI writer platform appeared first on IBA Group - USA.

]]>
Penetration Testing of Enterprise Absence Management System https://us.ibagroupit.com/cases/penetration-testing-enterprise-absence-management-system/ Thu, 01 Jun 2023 07:00:08 +0000 https://us.ibagroupit.com/?post_type=cases&p=9908 The post Penetration Testing of Enterprise Absence Management System appeared first on IBA Group - USA.

]]>

Challenges

Ensure that a delivered solution does not have security holes that could lead to customer’s sensitive data loss.

Identify vulnerabilities that could bring a high risk of product outage due to hacker attacks.

Solution highlights

Around 28 vulnerabilities were successfully eliminated from the software. 22 out of 28 vulnerabilities had either medium, high or critical threat levels.

Key Results

The customer’s audit was successfully passed after eliminating exploited vulnerabilities.

The post Penetration Testing of Enterprise Absence Management System appeared first on IBA Group - USA.

]]>
ETL Tool Visual Flow https://us.ibagroupit.com/solutions-and-products/visual-flow/ Fri, 13 Aug 2021 11:32:31 +0000 https://us.ibagroupit.com/?post_type=solutionsandproducts&p=7715 The post ETL Tool Visual Flow appeared first on IBA Group - USA.

]]>

Visual Flow

GET STARTED. Build ETL Pipelines Today

An ETL tool powered by Apache Spark on Kubernetes

We developed Visual Flow as a low-code solution for developers looking to leverage the ETL benefits of Spark without learning a programming language.

Try Visual Flow on AWS or GitHub

Apply for online demo

Visual Flow Logo

visual flow on AWS

visual flow on github

 

SPARKING ETL

A Low-Code Option for Building and Scheduling Spark Jobs.

We developed Visual Flow so you can jump-start Spark without writing a single line of code.

Until now, you’ve relied on a single-node ETL system. Now, your data is growing faster than your current engine can manage it. Distributed ETL engines like Spark offer tremendous upside. But when faced with coding jobs and pipelines manually, you run into a dilemma:

You don’t have time to learn coding languages
You can’t afford to hire external programmers
You don’t want to master an ETL tool, a build tool, and an orchestration tool
Your data continues to multiply out of control

The difficulty, expensiveness, and inefficiency of these options make Spark ETL seem impossible.

What You Will Get

By using Visual Flow, you will gain access to Spark ETL, transforming your data reserves into actionable wisdom.

Simplify ETL with a Drag-and-Drop Interface

Forget about Python, Scala, and Java, and embrace low- to no-code ETL on an easy-to-use interface.

Under-the-Hood Build and Orchestration

Visual Flow incorporates Argo Workflows, Kubernetes and the Spark processing engine. By mastering Visual Flow, you reap the benefits of all three software products.

Scale at the Speed of Data

Chain functions into jobs, then chain jobs into pipelines to transform raw data into Business Intelligence.

Visual Flow in Enterprise Infrastructure

How Visual Flow Works

Technical Features

Visual Flow is ETL, simplified. It allows users to leverage the scaling and efficiency capabilities of Spark ETL through an intuitive drag-and-drop interface.

Visual Flow includes:

  • Stage palette featuring nine discrete data transformation functions
  • Slack integration for user notifications
  • Support for data sources, including

● Relational DBs: IBM DB2, PostgreSQL, Oracle, MySQL, MSSQL, Amazon Redshift, ClickHouse

● NoSQL DBs: Elasticsearch, Cassandra, Redis, Mongo

● Cloud Object Storages: Amazon S3, IBM Cloud Object Storage

● Others: local files, dataframe

  • Ability to externally invoke Visual Flow pipelines using POST requests
  • GitHub and GitLab user directory support

Our Guiding Principles

Go Open Source

Visual Flow is an open source product that is based on other open source products.

Scale Optimally

Virtually unlimited horizontal scalability is provided by Spark.

Design Once — Run Anywhere

Once designed, deploy your ETL on any Kubernetes cluster anywhere.

Design Visually

Think of your data flows instead of the programming language syntax.

HOW TO USE VISUAL FLOW

Visual Flow is a low-cost, low-code, open-source product. Its portability, flexibility, and multi-cloud capability make it a powerful tool for a range of data projects.

  • Load Data from transactional sources into the data lake
  • Load application logs and structured files from a cloud object storage to the data lake
  • Cleanse and transform information inside the data lake
  • Load data from the data lake into the data warehouse

WHO VISUAL FLOW IS FOR

We developed Visual Flow for anyone looking for a convenient, intuitive ETL solution. Whether your data is multiplying faster than your single-server warehouse can process it, or you don’t have access to ETL coders for Spark, Visual Flow is for you.

  • Set up Visual Flow in under 30 minutes
  • Quickly design applications to transform large-scale, semi-structured datasets
  • Turn a messy backlog of data into uniform data lakes and warehouses
  • Intuitive, no-code interface
  • Standalone solution for intuitive ETL
  • Save money and time on training and/or new hires

Contact us

Please fill in the form to get in touch with us and share your details. Feel free to provide any additional information or specific questions you might have. Our team is committed to responding promptly and thoroughly.

Looking forward to hearing from you soon!

[contact-form-7]

The post ETL Tool Visual Flow appeared first on IBA Group - USA.

]]>
IBA Group Builds Global Enterprise Data Warehouse and Provides 24/7 Support https://us.ibagroupit.com/cases/data-warehouse-24-7-support/ Tue, 15 Dec 2020 12:08:07 +0000 https://us.ibagroupit.com/?post_type=cases&p=7073 The post IBA Group Builds Global Enterprise Data Warehouse and Provides 24/7 Support appeared first on IBA Group - USA.

]]>

Business Challenge

Development and support of Enterprise Data Warehouse, ETL processes and web application to process data.

Solution

IBA Group develops a financial planning and forecasting system for customer projects. A web application has been created for project managers to build financial forecasts. They also complete questionnaires and feedback forms used to assign statuses to the projects depending on their possible profitability/loss-making capability. After this data is processed, key performance indicators are generated. The reports generated by the system are submitted for verification to the company management.

A team of 30 developers and 10 technical support specialists is working on the project. They monitor ETL downloads around the clock and respond to user requests.

The system operates in five regions: USA, Asia, Europe, Japan and Latin America. Each region has individual ETL processes configured.

Result

The system generates reports for the company management to determine financial forecasts for the next quarter.

The post IBA Group Builds Global Enterprise Data Warehouse and Provides 24/7 Support appeared first on IBA Group - USA.

]]>
Logistics Company optimized BI applications and improved corporate data quality https://us.ibagroupit.com/cases/logistics-company-optimized-bi-applications-and-improved-corporate-data-quality/ Thu, 16 Apr 2020 11:50:41 +0000 https://us.ibagroupit.com/?post_type=cases&p=6615 The post Logistics Company optimized BI applications and improved corporate data quality appeared first on IBA Group - USA.

]]>

Business challenge

  • Conduct code audit, review of QlikView applications UI
  • Customize QlikView Applications
  • Implement Best Practices.

SOLUTIONS HIGHLIGHTS

IBA Group has developed an Enterprise Data Warehouse, multidimensional data arrays (OLAP cubes) and step-wise data import from various sources: transactional database, Sharepoint, Web services, SAP. Our experts analyzed the company’s processes, set up ETL processes and data cleansing, developed and customized SQL. They also introduced an abnormal event alert system for the ETL-system, configured information collection with the least load on database sources.

For the data warehouse of internal projects, we developed new models and processes for preparing data for QlikView. An IBA Group specialist checks the application for compliance with multiple criteria: design in accordance with corporate styling and coding standards, inclusion of certain visualization objects and proper optimization.

Auditors receive engagements to review the code and UI of applications. The customer receives reports in a standard format, and the developers receive revision lists. After correction, the application is subject to audit again. To speed up the process, the central office delivers some of the applications for refinement to experts from IBA Group.

Result

The Customer is continuously optimizing and improving the quality of QlikView applications and corporate data.

The post Logistics Company optimized BI applications and improved corporate data quality appeared first on IBA Group - USA.

]]>