• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
Codemotion Magazine

Codemotion Magazine

We code the future. Together

  • Discover
    • Live
    • Tech Communities
    • Hackathons
    • Coding Challenges
    • For Kids
  • Watch
    • Talks
    • Playlists
    • Edu Paths
  • Magazine
    • Backend
    • Frontend
    • AI/ML
    • DevOps
    • Dev Life
    • Soft Skills
    • Infographics
  • Talent
    • Discover Talent
    • Jobs
  • Partners
  • For Companies
Home » DevOps » Cloud » Migrating Data to the Cloud: A Practical Guide
Cloud

Migrating Data to the Cloud: A Practical Guide

This article shows the basic concepts of migrating datasets and offers a walkthrough for data migration from an on-premises data center to the Azure cloud.

November 22, 2021 by Gilad David Maayan

cloud migration
Table Of Contents
  1. What Is Data Migration?
  2. Types of Data Migration
  3. Data Migration Strategies
    • Big Bang Migration
    • Trickle Migration
  4. Data Migration Process
    • Planning
    • Migration Design
    • Migrating and Testing
  5. Real Life Example: Migrating Unstructured Data to Azure Storage
    • Discovery 
    • Assessment 
    • Migration 
  6. Conclusion

What Is Data Migration?

Data migration is a process designed and implemented for the purpose of transferring data between storage systems, computer systems or data formats. Organizations migrate data for various reasons. For example, when upgrading or replacing storage equipment or servers. 

This process is also implemented when shifting data over to third-party cloud providers, when consolidating websites, performing infrastructure maintenance and upgrading software. It is also used during data center relocation and company mergers.

In this article, I’ll explain the basic concepts of migrating datasets and present a general process for migrating your data. Finally, I’ll show a practical example – how to migrate data from an on-premises data center to the Azure cloud.

Transferring data to the cloud requires precise planning.

Types of Data Migration

Here are several data migration types:

  • Storage migration—involves the transfer of data from a certain storage repository to another. Storage migration can take place on-premises or in cloud environments. It is generally considered the most straightforward migration, but still requires a solid migration strategy. The target of storage migration in the cloud may be object storage, file storage, or block storage.
  • Database migration—the process of upgrading a database engine. It involves shifting an entire installed database of files into a new device. A database migration is typically more difficult to achieve than a storage migration. This is because you need to shift higher volumes of data, which might be formatted differently. It requires backing up your databases, detaching each database from the engine, and then migrating the files to the new engine. You can then restore the files to the new location and database.
  • Application migration—the process of shifting an entire software application from one location to another. This includes all application components, including databases, folders and installation files. It may require a combination of storage and database migrations. It may also require working with the application vendor to ensure the application functions properly post-migration.

Data Migration Strategies

Here are two commonly used data migration strategies.

Big Bang Migration

A big bang migration happens as a single event, during which live systems experience downtime. This happens because data goes through extract/transform/load (ETL) processing and moves to the new target database. This approach is usually associated with a greater level of risk if a failure occurs. 

Trickle Migration

A trickle migration is designed in phases, during which old and new systems run concurrently. This model eliminates operational interruptions and downtime. However, a trickle migration is often complex and time-consuming, which is why it requires careful planning.

Data Migration Process

The data migration process involves the following phases.

Planning

Start with evaluating your data assets and building a migration plan. You can break down the planning process into these steps:

  • Narrow the scope of migration—define the most critical data you need to migrate and filter out the excess. Consult with data users impacted by any proposed changes.
  • Assess the source and target systems—analyse the source system and the target system to evaluate how you can adapt the operational requirements of your current system to the new environment.
  • Set data standards—this is important for detecting issues during and after migration.
  • Set a timeline and budget—estimate the time and resources required to implement the migration process (this will differ according to the approach and scope of migration) and set deadlines and budgets.

Migration Design

Specify the rules, roles and acceptance criteria for migration and testing. Consider using data migration tools like Extract Transform and Load (ETL), and hire specialized developers or engineers to operate them. For unstructured data, consider treating data to reduce its size and complexity before migration – for example using image compression algorithms, video APIs or minification for text-based data. 

This phase involves creating data transition scripts and data mapping. A variety of developers, data engineers and business analysts may collaborate on the migration design. It is important to ensure the necessary tools are in place.

Migrating and Testing

Execute the migration and implement continuous testing. You can use a big bang approach to migration, which takes a day or two to transfer all the data but requires downtime, or a trickle approach, which is much slower but eliminates downtime and reduces the risk of critical failure.

Regardless of the approach you implement, testing should be baked into the design and execution of your migration. In a trickle approach, each portion of data should be tested when it is migrated, to ensure problems are fixed early. Apply frequent tests to ensure the retention of data integrity, both during and after the migration.

Real Life Example: Migrating Unstructured Data to Azure Storage

Microsoft Azure is the world’s second-largest cloud provider. Azure provides extensive migration resources that can help automate migration for both data and workloads – especially if your on-premises systems are based on Windows or Windows Server. Let’s see what the process looks like when migrating a large-scale, unstructured dataset to Azure storage.

Discovery 

This phase involves determining which sources you need to migrate. For example, you might decide to migrate your SMB shares, object namespaces or NFS exports. You can implement this phase manually or leverage automated tools.

Assessment 

This phase can help you understand which migration options are suitable for your scenario. Here are several steps you can implement for the assessment phase:

  • Choose a target storage service—the storage type you choose should suit the application and users accessing the data. To determine this aspect, you should do two types of assessments—a technical assessment to determine possible targets and a financial assessment to determine a cost-effective option. 
  • Select a migration method—Azure lets you choose between online and offline migration. The online migration method consumes network bandwidth when migrating data—either a public Internet connection or Azure ExpressRoute. Services without a public endpoint need to use a VPN over a public Internet connection. Offline migration requires the use of Azure Data Box devices. You can also combine the two methods.
  • Choose a suitable migration tool—you can choose from a wide range of third-party, first-party and open-source migration tools. Notable open-source options include AzCopy, xcopy, rsync and robocopy. You can combine native Azure Files capabilities with Azure File Sync to migrate from Windows file servers to Azure Files. Complex migrations might require the use of commercial tools.

Migration 

This phase involves moving the data. To accomplish an easier switchover, you should implement the migration phase several times. Here are the main steps of the migration phase:

  1. Initial migration—migrating all data from the source to a specific target. Typically, this step is responsible for moving the bulk of the data chosen for migration.
  2. Resync—migrating data that was changed after the initial migration step. If there are several changes, you can repeat this step multiple times. Running multiple resync operations can help you reduce the overall time it takes to complete the final step. 
  3. Final switchover—switching active usage from the source to the target data and then retiring the source.

Aspects that impact the duration of the migration for unstructured data include:

  • The total size of the data—the migration phase increases or decreases with the total.
  • File size distribution—the migration time increases or decreases with the average file size. To reduce the total migration time, you can archive small files within larger files. For example, .zip or .tar files.

Conclusion

In this article, I explained the basics of migrating storage to the cloud, which requires an understanding of data migration type, data strategies and the overall data migration process. I also provided an example of data migration phases, including:

  1. Discovery phase—determining exactly which sources you want to migrate.
  2. Assessment phase—choosing a suitable target storage option, migration method, and migration tool.
  3. Migration phase—moving most data during the initial migrating, cycling though multiple resync operations, and implementing the final switchover.

I hope this will be of help as you plan an efficient cloud migration that helps you smoothly and successfully move your data to the cloud.

Discover more about data migration in this video with Piet Van Dongen

Loading the player...
facebooktwitterlinkedinreddit
Share on:facebooktwitterlinkedinreddit

Tagged as:Azure Guidelines Migration

Top 7 QA Testing Best Practices to Ensure Overall Software Quality
Previous Post
Automating Cybersecurity With Machine Learning
Next Post

Related articles

  • Cloud-Native Development: About Data and the Use of Google Cloud Platform
  • 7 Programming Languages You Need to Learn to Succeed as A Cloud Professional
  • IBM Cloud CodeEngine: Beyond Serverless 1.0 for Large Organizations
  • How To Choose a Software Stack for a Cloud App
  • 10 Security Measures to Look Out for When Choosing a Cloud Vendor Contractor
  • Automating AWS Releases Without Cloud Expertise
  • Get to Know Cloud Run: Easy Deployment From the Cloud
  • How To Migrate and Manage Your Databases with Cloud Data Management
  • How To Guide: Integrating Spartacus Into a Multi-Project Angular Setup
  • Is Distributed Cloud the Future of Cloud Architecture?

Primary Sidebar

Learn new skills for 2023 with our Edu Paths!

Codemotion Edu Paths for 2023

Codemotion Talent · Remote Jobs

Java Developer & Technical Leader

S2E | Solutions2Enterprises
Full remote · Java · Spring · Docker · Kubernetes · Hibernate · SQL

AWS Cloud Architect

Kirey Group
Full remote · Amazon-Web-Services · Ansible · Hibernate · Kubernetes · Linux

Front-end Developer

Wolters Kluwer Italia
Full remote · Angular-2+ · AngularJS · TypeScript

Flutter Developer

3Bee
Full remote · Android · Flutter · Dart

Latest Articles

web accessibility standards, guidelines, WCAG

Implementing Web Accessibility in the Right Way

Web Developer

devops, devsecops, cibersecurity, testing

3 Data Breaches in Web Applications and Lessons Learned

Cybersecurity

The influence of Artificial Intelligence in HR

Devs Meet Ethics: the Influence of Artificial Intelligence In HR

AI/ML

google earth engine

What is Google Earth Engine and Why It’s Key For Sustainability Data Analysis

Data Science

Footer

  • Magazine
  • Events
  • Community
  • Learning
  • Kids
  • How to use our platform
  • Contact us
  • Become a Contributor
  • About Codemotion Magazine
  • How to run a meetup
  • Tools for virtual conferences

Follow us

  • Facebook
  • Twitter
  • LinkedIn
  • Instagram
  • YouTube
  • RSS

© Copyright Codemotion srl Via Marsala, 29/H, 00185 Roma P.IVA 12392791005 | Privacy policy | Terms and conditions

Follow us

  • Facebook
  • Twitter
  • LinkedIn
  • Instagram
  • RSS