A frequent issue With FDMEE processing is the time it takes to load data into a target system. One issue that significantly slows FDMEE processing is the use of LIKE mapping. If the number of LIKE Mappings gets large then the response time can be overly burdensome and may lead the user to wonder if there is an issue and the system has stopped processing.
The following is a method to increase the efficiency of LIKE mappings. It involves replacing a large number of LIKE mappings with a small number of LIKE mappings utilizing SQL.
First, let’s understand what FDMEE does with the like mappings. In order of the Rule Name given to the mapping, FDMEE then generates an SQL Update command the uses the Source Value in the where clause to set the target value. A separate SQL update command is generated and executed for each LIKE mapping
To illustrate, look at the following example:
FDMEE will generate five SQL commands similar to the following:
UPDATE TDATASEG_T
SET ACCOUNTX = 'T123456'
WHERE LOADID = 123
AND PARTITIONKEY = 111
AND CATKEY = 1
AND (ACCOUNTX IS NULL OR ACCOUNTX = '')
AND (ACCOUNT LIKE 'A1%')
AND PERIODKEY = '2017-06-30'
If there are 100 records in the TDATASEG_T table then 5 passes of the table or 500 records will be read to resolve all the LIKE mappings. This may not seem that large but scale the number of LIKE mappings to 1000 and the result will be 100,000 reads.
To make LIKE mappings more efficient, reducing the number of passes (LIKE statements) will reduce the processing time.
In the following example, the five LIKE mappings above are combined into one statement. This reduces the number of passes of the TDATASEG_T table from 5 to 1 and the number or Reads from 500 to 100.
Using the pencil icon to edit the script, the following can be used to replace the 5 LIKE mappings in the original set up:
Case
when ACCOUNT like'A1%' then 'T123456'
when ACCOUNT like 'A2%' then 'C124478'
when ACCOUNT like 'B1%' then 'M456329'
when ACCOUNT like 'C1%' then 'H777090'
else 'DEFAULT_ACCT
end
This will result in one SQL statement to map the accounts:
UPDATE TDATASEG_T
SET ACCOUNTX = Case
when ACCOUNT like'A1%' then 'T123456'
when ACCOUNT like 'A2%' then 'C124478'
when ACCOUNT like 'B1%' then 'M456329'
when ACCOUNT like 'C1%' then 'H777090'
else 'DEFAULT_ACCT'
end
WHERE LOADID = 123
AND PARTITIONKEY = 111
AND CATKEY = 1
AND (ACCOUNTX IS NULL OR ACCOUNTX = '')
AND PERIODKEY = '2017-06-30'
This will result in only one pass of the imported data to complete the mapping.
A few things to note when using this method are:
- For large amounts of mappings, multiple LIKES can still be used while significantly reducing the number.
- If the ELSE clause of the case statement is not used then if not match is found the NULL will result. This allows for multiple LIKES using SQL
- The SQL code must be valid for the database not FDMEE. Be careful with wildcards. The ESCAPE keyword and value may also be required.
- The SOURCE Value can be repeated. The Asterisk can appear multiple times. Remember that the RULE_NAME still must be unique and once a target value is set all other LIKE statements will be ignored for that record.
