Software Engineering Solutions for Business Challenges
Data Integration and Migration
Data integration consolidates information from disparate data stores into common, unified databases. With inexpensive computers and database software, departmental and individual user databases have sprung up everywhere. While providing increased productivity for one or a few people, they now present a number of problems. First, distributed databases often contain the same or overlapping information. This increases costs (entering changes into multiple systems) and decreases data quality (some systems don’t get updated or mistakes are made during the process). Second, it increases hardware, software, and administrative costs. Collectively, reducing these costs offers an excellent opportunity for savings.
Data migration is the process of transferring information from older data stores to a new database. The new database usually has a different data structure to meet new application requirements. Most legacy systems contain data which reflects operational requirements but no longer matches the original design. Typical examples are putting cell phone numbers into remarks fields, long addresses into too few address lines, international phone numbers wherever they will fit, and similar ways of squeezing data in where it wasn’t intended. Another common example is using one field, such as name suffix, for a different type of data, for example a prison number. We see this type of data anomaly when users need to accomplish a task (addresses printed out with a prison numbers attached) and the resulting business rule was not defined or implemented in the system.
Most data integration or migration tasks are complex, time consuming, and expensive. The existing data must be analyzed and understood, a new data structure must be defined, and a crosswalk developed to map data between the two. A program is written to transfer the data, using either a data integration utility or a programming language. That’s the easy part, then follows validating the new data and finding all the anomalies. The number of anomalies usually depends upon the age of the old system. In most cases, no one knows how many anomalies exist and therefore it is impossible to accurately determine just how long it will take to find and fix them. This is frustrating to managers and business owners who need to control costs.
While there are no magic solutions to these problems, there are ways to overcome them. Auburn SeeWolf brings an unusual range of experience to data integration and migration problems. Our business analysis skills allow us to define and understand the business process of both old and new applications. Our database design and administration experience allows us to understand the data structures and define crosswalks. Our system development experience enables us to select the most efficient off-the-shelf data integration tools or write custom data integration software when anomalies are too complex for simple solutions.
Data integration and migration can be a painful experience. This usually occurs when schedules and budgets don’t match the real effort needed to accurately and completely build the new data stores, or when the technical staff doesn’t have the experience needed to effectively deal with complex data anomalies. Auburn SeeWolf has the skills and experience to minimize the pain and bring these projects to a successful conclusion.
Copyright © Auburn SeeWolf 2009-2015