Why the need for data migration?
Not too long ago, database software were limited to few brands and options. To name a few of them we can say Oracle, MS SQL Server, MySQL, DB2 among few others. Oracle would be used for systems that needed very high performance and robustness. MySQL, would be used for web or small applications.
Licenses were a big factor, because depending on the database to be used and the need, the cost would substantially increase. Those were the days.
Today, things are different and changing even more day by day. Different types of DBMS are reaching the market every single day. The way, we view data is also different and with the rise of the cloud, the options are almost unlimited.
These changes are the main cause for a tremendous increase on the demand of data migration specialists.
Why companies are hiring so many professionals for data migration tasks?
As said previously, the DBMS options available nowadays are many more than what it was 10 years ago. In the past, depending on the size of the company and the data need, the options were pretty much limited.
As we already know, the least the competition, the higher the prices companies had to spend with all these companies like Oracle, Microsoft and so forth.
These companies are great and they do have fine systems, but the questions that are coming up into the executives from companies nowadays are, do we need all of that? Do we have better or similar systems with lower costs? Can we find better solutions for our problems with less expensive tools?
At the end of the day is all about money!
What is a data migration developer?
Data migration developer is professional who is capable of transferring data in different formats or DBMS systems from point A to point B. Notice that B can also be different formats from the source as well as different DBMS systems. This professional must also be able to handle what happens in between the migration and ensure that the data will behave as planned in the target system.
The majority of jobs, that I see today are for migration from Oracle or MS SQL Server to databases like PostgreSQL and MySQL. The obvious reason is the costs associated with the licenses that the proprietary ones are charging. It usually goes from a licensed one to an open source one. However, it can be found instances going the other way around.
There is also the cloud databases, where the cloud companies charge per usage. Even though those databases in the cloud can be any. The majority are either PostgreSQL or MySQL, again, due to no licenses needed for them.
In some cases, there are databases that were forked and modified by the cloud provider itself in a way that the provider is able to maintain the database running as optimal as possible. This is the case for Aurora in AWS or Big Query in GCP.
So, based on that, a must for developers are to know SQL (Structured Query Language). The developer must also be very flexible and agile in learning different syntaxes from various database systems. Which should not be very hard if ANSI patterns are used.
Is data migration a good career?
This is a great career path, however some considerations must be taken place before committing to it. The reason is because not always one single company needs data migration all the time. Data migration could in fact split in two different options:
- Data migration itself.
- Data conversion.
They could be synonyms, however, they could be interpreted differently based on a job description.
Data migration
It usually means that pipelines must be created to accommodate data from one active source to another active source. These are used in ETL and ELT processes and final destination is usually a data warehouse.
So, data warehouses are permanent jobs. As a data warehouse developer, we are set for one specific job for a long period of time if not forever, as long as the company or the professional lives. There are also a lot of opportunities as a data warehouse developer, engineer, integration and we can name it many more.
SQL knowledge is a must here. If the RDBMS is of the same name and version, this process can be much easier usually do not involve many tools to be done.
Data conversion
Contrary to data migration alone, this means that pipelines must be created to accommodate data from a soon to retire database to a new or ongoing active database. This one does not have to involve data warehouses, even though it could. It can simply be from one operational database to another operational database.
This usually happens when a company is merged with another company and the acquiring company wants to consolidated data from the acquired company.
Another situation would be migration to cloud. In the cloud, if the database vendor is the same, a simple back up and restore can do it. However, the trend now in the cloud is to move from commercial databases to open source databases. In that case, a more sophisticated approach must be taken.
There are many different approaches to tackle this type of situation. The more we can track and circle back the converted data from the new target to the old source, the least headache, we, developers, will have.
Some of the steps are very similar to migrating to data warehouses but the main difference is that in this case, normalization plays a huge role because both systems are supposedly normalized to an extent.
Data conversion steps.
A successful data conversion, requires few very important steps before, during and after. The correct set up can save hours and hours of work later on. It can also save frustration from clients and clients of the clients. This is not an easy process, but it can be simple.