Some Nerd Stuff – Recover from Store Corruption

I did get to be a nerd yesterday for a bit. I came into work and one of our customers had a SEV1 database issue. I got to work up a plan to fix the issue and do some talking to the customer. It was not super exciting, but It’s issues in my favorite part of Exchange. Below is a draft of an email I sent to the customer with steps.

COMPANY-X had a SEV1 outage last night; the ISCSI network connection was lost for all Managed Servers. The loss of ISCSI caused all of the Exchange databases to ungracefully shut down. The unexpected database shutdowns have caused visible corruption to the Exchange Databases. The most obvious signs of corruption are reports from users of attachments not being accessible in OWA. I’m presenting two options to COMPANY-X as follows:

 

·         Move Mailbox | Moving a mailbox in Exchange is the least user impacting method of repairing damage. When Exchange moves a mailbox to another database is runs integrity checks and verifies objects in the mailbox clearing up corruption. The user impact of the move operation is limited. During a move operation the mailbox being moved is unavailable for user access. In My Experience move mailbox will fix 75% of all corruption issues. The clean the corruption with the move mailbox Method the following steps should be taken:

1.       Create a new Database to replace a single database in a storage group

2.       Move all mailboxes the new database.

3.       Monitor Log volume space during the move to avoid filling the volume

4.       Remove the old database

5.       Repeat process until all databases are replaced

      

·         Repair the Database | Exchange ships with two tools to repair ESE databases; ESEUTIL and ISINTEG. ESEUTIL is a database level repair tool that looks at and repairs the database at the page level. ISINTEG is an application level tool that looks at, and repairs the database at the table and index level. Both tools need to be run on the database to repair the damage. During the repair process the database must be offline. When the database is off line none of the users with mailboxes on the database will have access to email. Based on the server size, and database size I estimate up to 6 hours of down time per database to complete the repairs. Best practice dictates we not leave repaired databases in production for an extended period of time. Once the databases are repaired and remounted we recommend that new databases are created and mailboxes are moved to them, like in the first option. In my Experience Repairing the Database will fix 95% of all corruption issues. To clean the corruption with the repair database method the following steps should be taken

1.       Take the database offline

2.       Make a file copy back of the backup

3.       Run ESEUTIL /P

4.       Run ISINTEG –fix

5.       At a convenient time Bring the database back online

6.       Create a new Database to replace a single database in a storage group

7.       Move all mailboxes the new database.

8.       Monitor Log volume space during the move to avoid filling the volume

9.       Remove the old database

10.   Repeat process until all databases are replaced

Related Posts with Thumbnails

About Kevinm