NEXT GEN BIG DATA WITH AZURE, DATABRICKS DELTA AND VALOREM REPLY
One of the largest tax preparation organizations in the US was facing depreciating, legacy solutions and restrictive and siloed data stores. Valorem Reply partnered with the client to accelerate their path to a more modern cloud infrastructure by building a state-of-the-art data and analytics platform using cutting-edge data lakehouse technology. Using Azure Data Services and Databricks Delta, the team designed and built an infinitely scalable foundation that will improve accuracy and timeliness of reporting, drive data quality, and enable data science research and machine learning activities to add more value to the business without breaking scale, even at the height of tax season. Ahead of its time, the resulting new data estate was developed with Spark cluster-based computing technology and delta live tables framework, to transform, ingest, and unpack extremely complex data into a centralized platform that will not only position the client’s Data Engineering team as leaders in the database marketplace, but also support the current and future data needs of their evolving business model.
With the foundation Valorem Reply has laid, we’re at a place where we are really able to make some big decisions with a really incredible tool at our fingertips. We’re already providing a window into data that the company has never had before. We built an infrastructure from scratch in a company that’s been around for a long time. That is not easy to do. So, it's a pretty incredible transformation in the amount of time that it’s been. Now we’re at place where we can literally do anything on the fly. It is pretty awesome.
We chose Valorem Reply because of their expertise in the Microsoft Azure space as well as the Databricks space. The Valorem team was able to take a loosely scoped project and come back to us with a clear understanding of what it meant and from the first meeting design solutions that fell in line with what I had envisioning almost to a T. The overall skillset is really high with Valorem. Though the first stage of our project is complete, we will keep a relationship with Valorem in place to continue to build out and innovate on the work done thus far. Just from a pure talent perspective, having Valorem around is not a bad idea.
CHALLENGES
- Depreciating legacy systems
- Unscalable, siloed on-prem data stores
- Restricted in-season analytics
- 5-year expense reduction and smart technology roadmap
SOLUTIONS
- Implement Azure-based data ecosystem.
- Enable data science and advanced analytics with a Databricks Data Lakehouse Platform.
- Improve velocity while executing business transformation with private preview Databricks Delta Live Tables approach.
RESULTS
- Data science enabled.
- Future scalability problems solved.
- Industry leading data and analytics platform.
- Query speed strengthened 10-100x.
- Foundational pattern to continue data migration and modernization.