Data Engineering

Provide transparency to your business and discover valuable insights out of data.

Why Customers Choose Akveo:

We combine BI technical expertise with business domain prowess.

We drive cost effective solutions, designed to match your goals, timeline, and budget.

We are experts at Data Engineering.

We have a proven track record of successfully delivered projects.

We Can Assist You With:

1

Data platform design

Let's build a state-of-the-art data and analytics platform design that your users would love.

2

Database Development

The use of custom database development best practises makes it possible to deliver maximum efficiency and generate more value.

3

Data Warehousing

Our cross-platform team will assist you in storing data regardless of your business requirements' complexity.

4

Business Intelligence

Our experience in implementing BI solutions will help you confidently initiate your projects.

5

IoT real time analytics, Big Data

We’re ready to help you with Big Data analytics and cutting-edge business intelligence tools to meet the needs of your customers.

6

Reporting and data visualization

We're ready to assist you with any data visualization challenges, be it health checking and maintaining your current data visualization solution, or designing and developing a new one from scratch.

7

ETL/ELT Development

We’ll be happy to assist you with the development and tech support of lightweight, flexible, and transparent ETL systems for extracting all important data from your sources.

8

Web Scraping

We can extract data from any website to identify and store integral information for future business use.

Our Workflow

Our team has the necessary expertise to help you at the stages of solution design, implementation, testing, and launch.

Technology Stack:

Amazon Redshift
Microsoft Flow
Stream Analytics
Azure Data Factory
Azure SQL
Oracle DB
Tableau
MySQL
Snowflake
Perl
Databricks
Spotfire
Power BI
Python
Azure

What Our Customers Say

Amy Curran, Velum Management

How We Built iOS, Android, and Web Apps for a Startup for the Price of One

Our business has expanded thanks to Akveo’s high-end services! Their contribution has exceeded our initial requirements. Being solution-oriented, Akveo's team establishes strong partnership relations to bring their clients tangible benefits. They always suggest innovative solutions, and remain open to any discussion.

Amy Curran

Principal Consultant, Velum Management

Key Features Our Projects Deliver

Contact us
1

Fully scalable data driven solutions that provide manufacturers with the production process insights to monitor, predict, and make the right decisions  at the right time

2

Modern cloud-based technology stack (Azure, AWS) that allows to save resources on maintenance

3

Automated process of gathering data

4

Real-time asset monitoring and immediate failure notifications

5

Detection of data anomalies and hints for fixing it

* Based on 100+ projects we delivered

Our Expertise in Implementing Data Engineering Solutions

How we did refactoring of an inefficient reporting solution

Customer- US-based developer of a messenger platform.
Key result - We’ve got a simple, straightforward and well-documented solution that can scale based on the current workload. This is cost effective as it only charges during processing and not when idle.

Challenges

The client had a reporting system for the messenger, but it was implemented without proper engineering approach:

  • each feature was implemented with no regard to the whole solution, cause there was no  thorough architecture;
  • it was impossible to find where specific feature had been implemented, as a whole bunch of tools and technologies were used: mix of cloud/on-premise, mix of languages (Python, Perl, sql, bash);
  • the documentation was missing.

Solution

Our team started from analyzing and documenting the existing solution and requests for improvements. The second step was to design a new solution's architecture and suggest a set of software tools which would cover required features, would be able to scale and cost-effective.

From scratch we implemented Data Warehouse, ETL process, Reports and Customer Portal. As well we enabled direct access to data in multi tenant Data Warehouse.

Technical Details

  • Data Warehouse size – 400Gb
  • ETL runs 1 time/day processing 1Gb
  • 150 tenants in single Data Warehouse
  • 3000 reports being generated and emailed daily

Toolset

  •  Snowflake for Data Warehouse
  •  Apache Airflow for ETL
  •  AWS EC2 for hosting Airflow and other tools
  •  AWS RDS for supporting relational database
  •  AWS S3 for File Staging and Storage for Generated Reports
  •  AWS SES for reports delivery via email
  • StimulSoft for reports generation

Team Composition:

  • Project Manager,
  • Data Architect, 
  • 4 Developers,
  • 1 DevOps, 
  • 1 QA, 
  • 1 BA

Key achievements:

We’ve got a simple, straightforward and well-documented solution that can scale based on the current workload. This is cost effective as it only charges during processing and not when idle.

 Our team successfully:

- analyzed and documented an old solution and requirements for the improvements;

- prepared a new solution proposal which included architecture, description of the mail features and suggested set of software tools;

- implemented the project and made a transition from the old solution to the new one.

How We Implemented Electronic Car Parts Catalogs

Customer- supplier of technology solutions in the automotive industry.
Key result - We successfully implemented Catalogs for 5 manufacturers, which helped the customer to save approximately $100 000 per on subscriptions.

Challenges

For years the client has been acquiring electronic parts catalogs spending $5000-50000$ per month for each catalog. Our team accepted a challenge to design and implement a solution which would generate and update the catalogs for 5 parts manufacturers from source data.

Solution

 Each manufacturer has their source data in their own format - text files, xml, excel, databases (.MDF) and even EBCDIC - the format used since the 50s on IBM mainframes.

The goal was to build ETL pipelines which would transform diverse source data and enable a single set of APIs to query all catalogs.

On Stage raw data is imported into MS SQL Database, cleaned and validated. Stage only contains a set of data being processed (it is cleared at the beginning of each ETL run).

Data Warehouse is a permanent data storage with all data sets integrated and normalized (not a canonical data warehouse).

Deliverable Database contains data with applied business rules and transformations required for enabling access by API. APIs are implemented as SQL procedures.

Technical Details

  • ETL runs 1 time/month for each manufacturer, processing time 1-8 hours
  • Largest Deliverable Database is 150Gb
  • APIs response time is 300ms (required a lot of optimization)
  • TeamCity is configured to run integrity checks, auto tests and build catalogs on full data sets
  • Auto tests cover ETL and all APIs, implemented on SpecFlow

Toolset

  •  On-premise Microsoft SQL Server for all databases
  •  Microsoft SSIS for ETL
  • TeamCity for Continuous Integration
  • SpecFlow for auto tests

Project Duration: 2.5 years

Team Composition:

  • Project Manager,
  • 6 Developers, 
  • 2 QA,
  • 1 BA,

Key achievements:

  • Customer saves approximately $100 000 per on subscriptions;
  • High quality of catalogs which saves resources on maintenance;
  • We enabled full access to catalogs and ability to update catalogs at any time.

Planning To Build Your Next-
Generation BI Solution?

Let us help you with that. Click the button below.

contact us