CASE STUDY
Legacy Infrastructure Retired, Time-to-Insight Cut by 60%: Google Cloud Migration Transforms Reporting for a Healthcare Analytics Firm
Key Results
| Time-to-Insight | Reduced by more than 60% after migration from on-premises infrastructure |
| Infrastructure Costs | 35% reduction in operating costs in year one |
| Scalability | Platform now scales to 10x current data volume without re-architecture |
| Manual Reports Eliminated | 12 weekly manual reports replaced with self-service Looker dashboards |
SERVICE AREA
INDUSTRY
CHALLENGE
On-premises infrastructure bottlenecking a business that sells speed and accuracy of insight
The client was providing medical analytics to hospitals and other health systems with real-time and accurate data analysis using on-premises technology that they had grown out of several years ago. Their “jobs” (data processes) were running at times over two hours when they should have been completed in minutes. As a result, analysts were having to find ways to get their work done within these limits (pre-aggregating data), running all “jobs” during non-working hours (usually overnight), and developing their own workarounds to make the report “technically correct”, but not truly accurate. Meanwhile, the cost of maintaining this existing infrastructure was not decreasing. The company’s IT team was spending too much time on maintaining old equipment, managing available resources, and ensuring that older methods of getting data into databases (ETL pipelines) continued to run. When their customers started asking for more detailed and current information, including transitioning away from weekly reports towards almost real-time dashboard reports, the gap between what the company’s current platform was able to provide and what the marketplace was expecting was growing quickly. Although leadership understood that migrating to a cloud-based technology was inevitable, they did not know which technology to migrate to or which type of architecture to use. They did understand that any migration that caused the application to go down or any data inconsistencies while the migration occurred would negatively impact their clients immediately.
SOLUTION
Phased Google Cloud migration with BigQuery-native analytics, Cloud Dataflow pipelines, and Looker self-service dashboards
The transition to Google Cloud was planned as a two-phase process. The first phase involved identifying a suitable location to migrate the current analytical and reporting applications (BigQuery) into the Google Cloud Platform (GCP). This decision was based upon GCP’s scalability with respect to large-scale analytics and the ability to integrate natively with BigQuery.
SCIGON’s goal in designing this migration strategy was to ensure that there would be no impact from a user perspective during the transition. To achieve this objective, SCIGON maintained both the old and new environments concurrently throughout the entire transition period. Each week, we validated that all data being produced by each system was equal in content before moving any client-facing application to utilize the new environment.
In addition to running both environments simultaneously during the migration, SCIGON replaced the client’s legacy ETL pipeline processes utilizing DataFlow for real-time and batch processing. We also migrated their data storage from a proprietary database (storage) to Cloud Storage and BigQuery for structured analytics workload. Additionally, we utilized Cloud Composer to implement an orchestration layer to manage all of these components and implemented a unified Identity & Access Management (IAM) layer using Google Cloud IAM to replace the client’s existing ad-hoc permission management methodology. The new methodology is a governed, auditable access control model that aligns with the compliance requirements associated with healthcare data.
From an analyst-facing perspective, SCIGON converted twelve manual weekly reports into Looker Dashboards, which connect directly to BigQuery. These dashboards provide clients with self-service access to their own data so they’ll have immediate access to the information they need without having to wait until the next scheduled report run.
As a result of the successful implementation of the cloud-native platform, our client now has a scalable and on-demand infrastructure solution that allows them to meet the increasing demands placed upon them by their growing client base and expanding data product portfolio, all without requiring another infrastructure cycle.
