Need to handle additional fields that fail data migration

Description

During testing with GK data, Jakub noted that some of the additional fields failed to migrate due problems like null values. Testing found for almost 2000000 additional fields migrated, there were 120 broken ones.

How do we handle these failures - can cleanup and manual migration occur after upgrade? Do we create a validation script to be run prior to upgrade to determine if problems are coming?


We have made the following commits to handle broken additional fields:

Environment

None

Activity

Show:
Jeff Brewster
December 11, 2010, 3:24 PM

Thanks Jakub. Couple of things I'd like to ask:
1. can you document where the QG log written by default, and the command you'd use to execute this parser for us to document in support/release notes.
2. Specific to GK, what is the query that should be used to find/fix the null prior to migration? I'd rather have GK fix their bad rows prior to migration if possible so that all additional fields work on first try in their test and production environments.

Jakub Sławiński
December 11, 2010, 3:39 PM

1. QG log simply is printed on stdout, so should be available in catalina.out. You have to trim this file to contain only messages printed during migration and run the following command:

awk -f qgmig_log_parser.awk catalina.out

2. You can find all NULL values using the following commands:

select * from customer_custom_field where FIELD_VALUE is NULL;
select * from account_custom_field where FIELD_VALUE is NULL;
select * from office_custom_field where FIELD_VALUE is NULL;
select * from personnel_custom_field where FIELD_VALUE is NULL;

You can fix those values using the following commands:

update customer_custom_field set FIELD_VALUE="" where FIELD_VALUE is NULL;
update account_custom_field set FIELD_VALUE="" where FIELD_VALUE is NULL;
update office_custom_field set FIELD_VALUE="" where FIELD_VALUE is NULL;
update personnel_custom_field set FIELD_VALUE="" where FIELD_VALUE is NULL;

Jeff Brewster
December 13, 2010, 10:51 PM

Jakub,
can you put the git reference in this issue where we now will take care of these null values as part of migration?

Jakub Sławiński
December 14, 2010, 6:30 AM

Fix for null values has been committed in ebb076e9db6a790d78ca71d02a73459975b298e5

Lukasz Chudy
December 15, 2010, 2:04 PM

Verified on 15.12.2010
Bug is fixed properly.

Commit: 391a3c860b0bedee237229717562cc58c131a7f0
Build Number: hudson-head-1.7.x-370
Branch Name: origin/1.7.x

Assignee

Jakub Sławiński

Reporter

Jeff Brewster

Labels

None

URL

None

Story Points

0.5

Team

Core

Scheduled For

None

Epic

Notify

None

productboard URL

None

Man Day Estimate

None

Components

Affects versions

Priority

Major
Configure