In the payments domain, transaction processing (i.e., authorization, clearing and settlement) is supported by multiple applications associated with compliance, enterprise data warehousing, loyalty, merchant services, and risk mitigation. Traditionally, most of these applications, along with the transaction processing application itself, have been operated on mainframes. Most data migration projects involving these key appications are either open systems (e.g., Unix) or big data platforms (e.g., HDFS, NO SQL, NEW SQL, etc.). These migrations have been undertaken to lower overhead costs, improve performance, address system consolidation, enhance system performance and gain a competitive edge in the marketplace.

All the systems mentioned above must meet the strict requirements of payment industry compliance. Therefore, when choosing the new platform to be used to store data, a key attribute to review thoroughly concerns ability to meet regulatory standards. Also, any down-time that maybe involved while switching to the new platform must be carefully considered based on the criticality of relevant systems. If data migration impacts systems that support real-time risk evaluation or real-time updates in loyalty systems, special care needs to be taken to minimize down-time. Running systems in parallel during the migration can address concerns over down-time allowing for the completion of the project before going live with the new platform. Lastly, if an organization utilizes multi-data center application hosting, the new data platform will need to have cross data center synchronization features.

RS Software has successfully completed numerous data migration projects using frameworks designed specifically for clients working in the payments industry. The diagram below represents the main components of the RS Software framework:

• Data Migration Specific Study
  Studies of several key elements are conducted in order to identify the variables that might increase risk to the project. These studies help project and program management teams size related information and create project plans.
  • Data Migration Purpose Identification: Though it may seem basic, identifying and clearly stating the reasons an organization has decided to initiate a data migration project forms the most important element in the framework. All other elements, stages and steps in the project will be based on these reasons. Generally, reasons for the projects may include the desire to decrease overhead, increase operational efficiency, consolidate a multi-platform data housing strategy to a single platform, improved functionality within the platform and/or gain a competitive advantage.
  • Data Migration Type Identification: There are different types of data migration that vary from simple to complex. The project can be as simple as single-source to single-target or as complex as multi-source to single-target or multi-source to multi-target.
  • System Type Identification: The number of steps involved in a migration varies based on a system’s data ingestion process. If the system is involved in providing OLTP services and data gets populated using web or desktop applications, a one-time data migration step will suffice. However, if the project involves an OLAP system, it will need enhancement to ETL scripts along with one-time migration scripts. Also, if data migration involves real-time or near-real-time applications, additional steps need to be taken to ensure minimum to zero downtime of the applications during the project.
• Data Profiling and Data Quality Measurement of Source Systems: RS Software recommends that the health of the data within the source systems be evaluated using a variety of quality metrics. This eliminates the proliferation of aged, incorrect data and provides a baseline to validate the data captured by the target system.
• Target System Evaluation: It is crucial to identify all features that are currently supported by source systems to insure that the possible target systems evaluated provide the same features.
• Downtime to Switch: This is another important factor that helps determine what migration tools and approaches are used in the project. If an application requires zero downtime, there are tools from vendors like Oracle, IBM and Golden Gate that are built to support zero downtime migrations.

• Data Migration Process Activities – Building the process that will support the data migration initiative is the most time consuming of the steps that must be taken during the project.
  • Data Migration Blueprint: High-level representation of the data migration process that is accepted by all project stakeholders.
  • Data Migration Technology Stack: The evaluation of the tools to be used in the data migration process. Some organizations do this step as a pre-processing component. RS Software has extensive experience in using the open source and licensed stacks.
  • Conflict Resolution: This step will be primarily for multi-source to single-source data migration. All stakeholders and business owners should agree on each step that is involved in resolving conflicts that may arise.
  • Data Lineage: Data lineage also will need to be addressed in the migration as required based on the current metadata management system.
  • Data Security: It is mandatory to involve data governance and security team members in the project to make sure all data security standards are followed or enhanced as per established guidelines.
  • Documentation Requirements: The level and extent of documents as required by an organization’s standards must be defined and incorporated.
  • Test Cases for Data Migration Validation: Test cases are identified based on the elements in pre-processing and conflict resolution. Test cases should include:
    • Data Loss, Corruption and Integrity: These are the primary concerns in any data migration project. The case studies should address basic data validation, mitigation censoring and truncation.
    • Performance: Test cases should address performance-specific testing. Existing test cases can be utilized if the application provides OLTP services.
    • Semantic Risk: These can be automated test cases or random samplings with structure identification and outlier detection.
    • Data Imputing: Based on past experience, it is crucial that previous platforms need some data imputing because of application related issues or due to multi-platform to single-platform consolidation. Data profiling test cases should cover test cases for data imputing.
  • Development: Using all the elements above, development involves what is required for data migration as well as enhancements to existing data ingestion script.

• Testing – Testing and sign-off on all approved test cases.
• Performance Measurement: This step is mandatory in most of the data migrations to confirm that performance measurements meet the defined thresholds and SLAs.