top of page

Demystify Exchange-mailbox migration




My very fist Microsoft Exchange version was V4.0 and my first Exchange Migration was from V4.0 to Microsoft Exchange 2000. It was very painful and complicated migration. There was no migration documentation or something like this and no MVP blogposts on the internet. You could just relay on your experience and knowledge of other colleagues. We needed to change credentials from every mailbox, export all emails to PST, import all PST files to the new Exchange-Server and configure Outlook manually on every computer for every employee. Since then, I am continuously migrating mailboxes to the next version of Microsoft Exchange. Since 1996 mailbox migration has been improved, but still very complex. In this article I will try to deep dive into several migration scenarios and explain Microsoft Exchange On-Prem migration to Microsoft Online.

 


 

Migration principals

Before I will start with deep dive in to Exchange-Mail migration, I would like to explain some terms and give you a “Start-Up” information, so that you can understand this article in full color range.

What is mail migration?

Mail-Migration is not only migration of email objects from Source to Destination, but also a change of communication and collaboration infrastructure for your company and have a big impact on productivity and business continuity. Any kind of migration may have an impact on:

- Mailbox access

- Mailbox permissions

- Mailflow and mailrelay

- Calender availability

- Distributiongroup- and Securitygroupmembership

- Resource and Sharedmailbox / Functionionalmailbox access

- Addressbook access and membership

- Permission delegation and collaboration

- Email security, like antispam / antiphishing, antivirus and email encryption (conditional access)

So, if you are planning an email migration, think about above mentioned points and how to deal with it.


Which mail migration exists and which are covered in this article?



In this article I will explain only Exchange to Exchange migration. So, I will not explain any 3rd party to Exchange-Migration, because there are tons of tools on the market like QUEST which you can use for it.

Exchange to Exchange scenarios.

Following migration scenarios are covered in this article.



In scenario A we have one Exchange-Organization and one data center. All Exchange Servers are in the same data center. You need to move mailboxes from Exchange-Server B to Exchange-Server A.



In scenario B we have one Exchange-Organization in two data centers. You need to move mailboxes from Exchange-Servers located in data center 1 to Exchange-Servers located in data center 2.




In scenario C we have one Exchange-Organization in two data centers. We are using load-balancers for better quality of service.

It doesn’t matter how many load-balancers do you have in each data center, architecture is almost the same.

You need to move mailboxes from Exchange-Servers located in data center 1 to Exchange-Servers located in data center 2.




In scenario D we have two Exchange-Organizations and two data centers. We have two different forests and need to move mailboxes from one forest to another. This type of migration is called “Forest to Forest” migration and requires additional preparation of target and source forests. Check documentation of this feature here. We will focus only on mailbox moves and not on “Forrest to Forrest” deployment.



In scenario E we have one data center and one Exchange-Organization. We have an Exchange-Hybrid configuration (doesn’t matter if Centralized mailflow, Hybrid-mailflow, or HMA). We are using separate “Exchange-Hybrid” servers only for Client-Access / Hybrid-Configuration-Wizard and have separate “Maibox Servers” which are hosting mailbox-databases in a DAG or without.

All user objects are synchronized via Azure-AD-Sync to AzureAD.

Important: We have only 1 data center, all “Exchange to Exchange” connections are running in the same LAN. You need to move mailboxes from Exchange Mailbox-Servers to Exchange Online.


This is the most complicated scenario which is used in enterprise environment with more than one data center “multiple sites”. We have two data centers and one Exchange-Organization. DC1 is our “Headquarter” and DC2 is our “Branch office”. Between both DCs is a “WAN” connection like MPLs, or VPN tunnel. DC2 has no internet connection, the whole traffic is routed through DC1 via “WAN” link. We have an Exchange-Hybrid configuration (doesn’t matter if Centralized mailflow, Hybrid-mailflow, or HMA). In DC1 we have “Exchange-Hybrid” servers located only for Client-Access / Hybrid-Configuration-Wizard and in DC2 our “Maibox Servers” which are hosting mailbox-databases in a DAG or without.

All user objects are synchronized via Azure-AD-Sync to AzureAD. AzureAD sync is located in DC1. You need to move mailboxes from Exchange Mailbox-Servers over “WAN” connection to Exchange Online.


If you have a scenario which is totally different, as mentioned above, just let me know, I would really appreciate to get more ideas and increase the quality of this article.

MRS proxy


It is a part of Exchange-WebServices and is responsible for Mail-Box migrations. This service is based on EWS (Exchange Webservices) and is handled in the same way as all other EWS features like “HMA”. You can find MRS proxy documentation here.

Important: Exchange-Server is proxying MRS traffic internal via port 444 and only initial Moverequestis using 443. Client/Access Server and Mailbox / Source Server need to have the same External URL configured in EWS.

Network configuration

MRS traffic can be used internal and external.

- External traffic means: if destination server is in a different forest or belongs to Exchange-Online (scenario D,E and F). In this case following ports are required to be opened on your external FW in DC1.

Pot

I

Port

IP

Explanation

​443

Destination-Server-IP

Inbound HTTPS

443

Client / Access Server

​Outbound HTTPS

443

Mailbox / Source Server

Outbound HTTPS

- Internal traffic: means if Client / Access Server and Mailbox / Source Server are in the same forest and can route all traffic using private IP ranges.

Microsoft recommends to open all ports for internal traffic between Exchange-Servers in your Exchange-Organization.

Following Windows-ports are listed in Microsoft Documentation


Inbound

Protocol

Port 135 between all Exchange Servers

RPC

Port 445 between all Exchange Servers

SMB

Port 443 between all Exchange Servers

HTTPS

Port 5985 between all Exchange Servers

WinRM

Port 3389 (RDP optional)

RDP

Port 443 from user lan

HTTPS

Port 808 for remote move requests


Outbound

Protocol

Port 53 to DC

DNS

Port 389 to DC

LDAP

Port 636 to DC

LDAP SSL

Port 25 to Smart-Host

SMTP

Port 443 between all Exchange Servers

HTTPS

Port 444 between all Exchange Servers

HTTPS

random port number between 1024 - 65535

High ports

random port number between 49152 – 65535

High ports

Port 808 for remote move requests


Authentication



During the migration, authentication flow is based on Source (forest) and destination (forest) credentials. Move requests are always created on Destination-Server and need to be monitored right there.

- Source credentials need to be able to connect to Exchange-Destination-Server and manage mailboxes (Recipient Manager)

- Destination credentials need to be able to connect to Exchange-Source-Server and manage mailboxes (Recipient Manager)


Logging

All log files are located in Microsoft Exchange-Server installation directory.

You can find EWS troubleshooting guide here.


HTTPProxy logs for EWS:

%ExchangeInstallPath%Logging\HttpProxy\Ews

Example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

The name of the HTTPProxy logs contains the date and hour starting to log, for example HttpProxy_2019093014-10.LOG (10th log from Sept 30, 2019, starting hour 14:00 UTC)


Migration batches

This feature is a part of Exchange-Online. It allows you to create single or bulk migrations without any PowerShell / Scripting knowhow. You can create, monitor and manage your mailbox migration simple from the “New” Exchange-Online-Admincenter.


Creation



You can find the function “Add Migration batch” directly in Migration hive.



You can migrate to Exchange-Online and from Exchange-Online to your On-Prem migration endpoint.


You can select following migration types.


Only “Remote move migration” is related to this article.



After you have selected the right migration-endpoint you can select User/s who should be migrated or provide a CSV file with a list of “EmailAddresses” of users.



You can find exact documentation how to use CSV files here.

Important: You don’t need any remote credentials for this task. Exchange-Admin center is using current configured credentials of your migration-endpoint. So, make sure that your MRS-Endpoint credentials are still valid.

For validation of current MRS-endpoint credentials use following command

Get-MigrationEndpoint | fl

Find documentation for this CMDlet here.


Monitoring


After you have created a migration batch you can use Migration-Report located in Monitoring hive. It will provide you a list of running batches and migration status in a PowerBi report.

Exact documentation you can find here.


Management

You can manage migration-batches and single user moves directly in the new Exchange-Admin-Center.

Exact documentation you can find here.


PowerShell

Additionally, to the New Exchange-Admin-Center you can handle all previous steps by using Exchange-Online-Powershell.

Important: You need to install ExchangeOnlineManagement powershell module to manage migration batches.

How to get migration batch information?

You can get migration batch information by running following command:

Get-MigrationBatch 

For a specific migration-batch you can use “-identity” property

Get-MigrationBatch -Identity 'MigrationService:2ndWave'


How to get single mailbox migration status?

You can get single mailbox migration status by running following commands:

Get-MigrationBatch -Identity 'MigrationService:2ndWave' | Get-MigrationUser

This scriptline delivers a basic status for all single mailbox migrations in the batch.

Get-MigrationUser -BatchId 'MigrationService:2ndWave' | Get-MigrationUserStatistics

This scriptline delivers a full overview of all single mailbox migrations in the batch.


Here you can find all CMDlets related to move-and-migration.


Move-Requests

Exchange Moverequest is a feature in Microsoft Exchange Server that allows administrators to move mailboxes between Exchange servers or databases. It is commonly used during server maintenance, upgrades, or when consolidating resources.

The Moverequestcan be performed within the same Exchange organization or even across different Active Directory forests.

Moverequestis the main mechanism for transferring mailboxes from Source to Destination server.

Important: If you are migrating from Exchange-Onprem to Exchange-Online and you cannot use “migration-batches” feature, because of different reasons, you can create move-requests and order them in batches manually.

If you need additional information, read Microsoft Documentation.


Creation

You can create move-requests On-Prem in the same ways as in the Cloud.

Navigate ECP of your Exchange-Server or Exchange-Online ECP (https://outlook.office365.com/ecp/)


Navigate “recipients” hive and go to “migration”


Important: Navigate ECP on your Destination-Server and not on your Mailbox / Source-Server. In some cases, Move-Requests are visible only on Destination-Server.

I wrote a Create-MoveRequests function for bulk moverequest creation to speed up the process.


This is a sample call of this function.

Create-MoveRequests -batchname '2ndBatch'-userlistlocation 'C:\temp\userlist.txt' -completeafterdays '5' 


Monitoring

If you want to see status of you moverequest just simply select one of your moverequests directly in ECP.


Important: If you create moverequests in Exchange-Online you can see them only by using PowerShell

You can export status of all move-requests with following commands

Get all moverequests:

Get-MoveRequest

Get all moverequests of a specific batch:

Get-MoveRequest-BatchName '2ndbatch' 

Get extended moverequests information of a specific batch:

Get-MoveRequest-BatchName '2ndbatch'| Get-MoveRequestStatistics 

Export extended moverequests information of a specific batch to a CSV file:

Get-MoveRequest-BatchName '2ndbatch'| Get-MoveRequestStatistics | ConvertTo-Csv-Delimiter ';'| Out-File-FilePath 'C:\temp\moverequest-status.csv' 


I wrote Export-MoveRequestStatistics function for exporting of move-request statistics and failure reporting of current migration / batches.


This is a sample call of this function.

Export-MoveRequestStatistics 

Or use “Batchname” property.

Export-MoveRequestStatistics -batchname '2ndbatch' 


Management

For moverequest management you can use ECP es mentioned before and PowerShell. ECP give you limited possibilities to manage moverequests, so if you need some advanced actions you need to use PowerShell.

PowerShell gives you following management possibilities:

- Get-MoveRequest: gives you moverequest information.

- Set-MoveRequest: configure additional settings to existing moverequest

- Remove-Moverequest: remove existing moverequest

- Resume-MoveRequest: will resume a failed or suspended moverequest

- Suspend-MoveRequest: will suspend a moverequest “InProgress” status

- Get-MoveRequestStatistics: gives you advanced information to a moverequest


Bulk removal of moverequests

For a mass moverequest removal I wrote Cancel-Moverequests function, which can remove all moverequests of a batch, or a specific user list.


This is a sample call of this function.

Cancel-Moverequests -batchname '2ndwave' 

Or you can use a list a Emailaddresses for removal (.CSV or TXT)

Cancel-Moverequests -userlistlocation 'C:\temp\userlist.txt' 

Failed moverequests bulk removal

For a mass mailbox removal of failed moverequests I wrote Cancel-FailedMoverequests function, which can remove all failed moverequests and generate a log file of all canceled moverequests.


This is a sample call of this function.

Cancel-FailedMoverequests 

Or you can use a list a Emailaddresses for removal (.CSV or TXT)

Cancel-FailedMoverequests -userlistlocation 'C:\temp\userlist.txt' 

Bulk configuration of batchname for a bunch of moverequests

For setting up a Batchname for a bulk of moverequests I wrote Set-Batchname function, which is able to set a batchname for a specific, already existing, batch.


This is a sample call of this function.

Set-BatchName-batchname '2ndwave'-newbatchname '3ndwave' 

Moverequest Status


Moverequests have a lot of different status information and switches. You can use Get-Moverequest CMDlet to get basic information and if you need extended information, you need to use Get-MoveRequestStatistics.


Basic status information

If you run Get-MoveRequest command in ExchangeOnlineManagement shell you can get following Status information:

- AutoSuspended

- Completed

- CompletedWithSkippedItems

- CompletedWithWarning

- CompletionInProgress

- Failed

- InProgress

- Queued

- Retrying

- Suspended

- Synced


Advanced status information

If you need extended information for a moverequest you need to use Get-MoveRequestStatistics. This CMDlet delivers a full range of information related to moverequest and mailbox migration.


Following information can be retrieved by running this CMDlet:

StatusDetails

  • AutoSuspended

  • Canceled

  • Cleanup

  • Completed

  • CompletedWithWarnings

  • Completion

  • CopyingMessages

  • CreatingFolderHierarchy

  • CreatingInitialSyncCheckpoint

  • CreatingMailbox

  • DataReplicationWait

  • Failed

  • FailedBadItemLimit

  • FailedMAPI

  • FailedNetwork

  • FailedOther

  • FailedStallDueToCI

  • FailedStallDueToHA

  • Finalization

  • IncrementalSync

  • InitializingMove

  • InitialSeeding

  • InProgress

  • LoadingMessages

  • MDBOffline

  • NetworkFailure

  • OverallMove

  • ProxyBackoff

  • Queued

  • Relinquished

  • Stalled

  • StalledDueToCI

  • StalledDueToHA

  • Suspended

  • TransientFailure

  • ADUpdate

  • StalledDueToMailboxLock

  • StalledDueToReadThrottle

  • StalledDueToWriteThrottle

  • StalledDueToReadCpu

  • StalledDueToWriteCpu

  • StalledDueToReadUnknown

  • StalledDueToWriteUnknown

SyncStage

  • Cleanup

  • CopyingMessages

  • CreatingFolderHierarchy

  • CreatingInitialSyncCheckpoint

  • CreatingMailbox

  • FinalIncrementalSync

  • IncrementalSync

  • LoadingMessages

  • MailboxCreated

  • SyncFinished


For additional information read this Microsoft article.


Synchronization


Email synchronization and active mailbox move/migration is running only if the status of moverequest is InProgress. So, if your moverequests are Failed, or Suspended, there is almost no traffic between source and destination server.

In the scenario E and F, synchronization traffic is using port 444 and not 443.

You can resume Failed or Suspended moverequests by running Resume-MoveRequest.

If you resume moverequests, synchronization is starting at that point where it stopped, as long moverequest is persisting.


Completion date


Every moverequest has a deadline when it should be finished.

- If the mailbox is synchronized before the deadline it will stay on status Synced short time before reaching the deadline. Before deadline is reached it will start with a delta-sync of current mailbox and then complete the moverequest.

- If AutoSuspend preference is set to $true or you have no CompleteAfter parameter configured, moverequest will sync until 95% and then stop any activities, as long CompleteAfter parameters are not changed.

- If you have configured CompleteAfter parameter, moverequests will stop at 95% and try to run deltasync every 24 hours until the CompleteAfter data has been reached and then moverequest will be completed automatically.

- If you run Resume-MoveRequest without setting up the CompleteAfter parameter, it will just start deltasync for all mail items and stop at 95%

Default value of CompleteAfter is around 100 years if you don’t change this parameter.

Bulk CompleteAfterDate configuration

For setting up a CompleteAfter date for a bunch of moverequests I wrote Set-CompleteAfterDate function, which is able to set CompleteAfter date for a specific batch or a list of users for specific amount of days.


This is a sample call of this function.

Set-CompleteAfterDate -batchname '2ndWave'-amountofdays 5 

Or you can use a list of EmailAddresses of moverequests which should be completed after specific amount of days.

Set-CompleteAfterDate -userlistlocation 'C:\temp\userlist.txt' -amountofdays 5 

Troubleshooting



A standard migration of mailboxes from Exchange A to Exchange B doesn’t require any troubleshooting effort, except “BadItemsLimit” or some other ‘historical’ errors.

Microsoft is providing a convenient collection of troubleshooting guide for MailMigration here.

I will explain migration troubleshooting on example of most complex architecture in Scenario F.


A lot of ERRORS



In a big enterprise environment, you have a lot of different components across the infrastructure which may cause an error. So before starting with reverse-engineering of all components, you should create a troubleshooting plan.


The Troubleshooting plan.

This is a summary of strategical steps for troubleshooting of Exchange – to – Exchange mail migration.


Step One

Validate your environment according Migration Principals in this article. Check all requirements and considerations.


Step two

Minimize the complexity of your deployment.

- Create host entries for all Exchange-Servers manually, to avoid DNS issues.

- Directly use Exchange IP Addresses and hostnames in your host files, to void any load balancer misconfiguration.

- Enable ICMP protocol between all Exchange-Servers and responsible Domain-Controller for each site.


Step three



Validate if internal migration is working.


- Migrate a mailbox from Mailbox / Source Server to Client / Access server.

- Migrate a mailbox from Client / Access Server to Destination server.


Step four

After you have completed all three previous steps, you can start with troubleshooting and analyze Failure Events of your moverequests.

If there were some changes in your infrastructure / environment and your migration started to fail, just, ROLL BACK ALL CHANGES.

Failure Events

During the migration you may face a lot of Failure events and errors. All of them we can summarize in two Group:

- Deployment Errors

- Performance Errors

Deployment Errors

This kind of errors are related only to your deployment of exchange infrastructure.

If you are trying to create a moverequest and it is failing immediately, and it was never working before you need to restart with “The Troubleshooting Plan”.


Performance Errors

Once you have completed first three steps of our Troubleshooting Plan and you were able to migrate mailboxes from Mailbox / Source Server to Destination Server, all requirements and prerequisites are achieved, and we can start with troubleshooting of failure events.


How to get failure events from moverequests


Failure information of a moverequest is included in Get-MoveRequest and Get-MoveRequestStatistics CMDlets.

You can try both according to your requirements.

You can run following commands:

get-moverequest-Status failed 

To get all “failed” moverequests

get-moverequest-Status failed | Get-MoveRequestStatistics | select identity, FailureType, FailureSide, Message, FailureTimestamp 

To get all failure information from your failed moverequests.


Use -includereport property


Get-MoveRequestStatistics -Identity mccloud@mccloud.cloud -IncludeReport 

For detailed failure protocol. It will include all issues and retries of your moverequest.


Important: MRS proxy is trying to restart every moverequest only 60 times, after that, the status is changing to Failed.

If have found this article from Microsoft that, there is a way how to change the MaxRetry value. In our case this change, had no effect (Exchange 2016).




MRS proxy


As explained in principals, MRS is the same as EWS webservice on every exchange. You can follow this checklist to validate EWS functionality.


EWS checklist

Restart IIS WebService

IISRESET /noforce

Check EWS availability and health

https://mccexch01.mccloud.cloud/ews/healthcheck.htm
the reply should be 200 OK

Restart EWS application pool

Restart-WebAppPool -Name " MSExchangeServicesAppPool"
!!! DO NOT DO IT during business hours, it has impact on user connectivity!!!!

Check IIS Certificate


Use CURL command to check HTTPS connection to all your EWS webservices internal and external


MRS performance settings


MRS replication service is transferring email data from Mailbox / Source server not in realtime, or bit by bit. It is using data packets of “default” 7168KB and trying to transfer it at least every 20 minutes. If the transfer is not possible because of bad latency or packet lost status details is returning following error codes:

o CommunicationErrorPermanentException

o MailboxIsNotInExpectedMDBPermanentException

o SourceMailboxAlreadyBeingMovedPermanentException

We have solved this issue by changing of following values:


ExportBufferSizeOverrideKB does not persist by default, so you need to add this line and change the value to desired KB value. After this change, Mailbox / Source Servers are using smaller packets which has enormous performance benefit.


Additionally, we have changed DataImportTimeout to 5 mins.


After this change your exchange-servers may consume more CPU utilization, so make sure that you have enough CPU resources.

Important: you don’t need to reboot your servers or restart IIS service, to have an effect. It is starting to work after you have saved the file!

This is the only Microsoft article which describes this issue.

Additional MRS proxy performance settings you can find here.


Waiting for approval

Some moverequests may stuck on status Synced and waiting for approval.

Just approve them by using following command.


Set-MoveRequest-Identity 'cloudy@mccloud.cloud' -SkippedItemApprovalTime $(get-date).ToUniversalTime() 
Resume-MoveRequest -Identity 'cloudy@mccloud.cloud' 


Network bandwidth


Network performance is very important factor during mailbox migration. You need to calculate network throughput before you are starting with migration planning. Calculate with the least bandwidth and not with maximum speed. Validate if some other applications are using your uplink connection.