Being a credit reference bureau, Transunion crunches A LOT of data.
Transunion already had a system that loaded the data written by a very brilliant programmer. He had defined his own validation language and used cool things like Scala and Kafka. Only problem was that he had to leave on short notice. and attempts to modify it had made it a bit unreliable.
The task was to redesign the system with the following goals;
1. Keep the custom validation language.
2. Maximum reliability. The manager wanted to start getting home in time for supper.
3. Make it configurable (Transunion Kenya supports 4 other countries on the continent, the previous system only served Kenya)
With guidance from the senior dev, we settled on Spring Batch for the following reasons
1. Dependency Injection - write once, set up different contexts
2. Spring Batch handles the concurrency and is wickedly fast
3. Other cool Spring tricks like cron like job scheduling
The main challenge was ensure thread safety.