“Llama is free, why are we paying a model provider?” It is a fair question in a lender’s boardroom, and it has a specific answer. Free to download is not free to run, and a mortgage shop is the last place with spare ML infrastructure lying around.
The GPU bill. Running a large open-weights model with sub-two-second latency for a retrieval application takes serious VRAM, which means multiple high-end datacenter GPUs. On-premise, that is capital tied up in silicon, plus power and cooling. In the cloud it is operating cost that keeps running whether utilization is 100 percent or 10 percent.
The MLOps you now own. Call a hosted model API and batching, quantization, KV caching, and serving throughput are someone else’s problem. Self-host and they are yours, along with the team to run inference servers, handle autoscaling, keep availability up, and manage version updates. Most lenders do not have that team, and hiring it is harder than hiring underwriters.
The patching treadmill. Open models and the container stacks under them ship vulnerabilities. In a managed service the provider patches them. Self-hosted, your InfoSec team scans your AI containers for CVEs on an ongoing basis, on top of everything it already carries.
When self-hosting does make sense
- Data gravity. Borrower NPI so sensitive it cannot leave the lender’s VPC under GLBA. This one holds.
- Volume. Millions of tokens a day, where per-call API cost exceeds the fixed cost of owning the hardware.
- A narrow, tuned task. A small model fine-tuned to classify 600+ mortgage document types, or to parse a specific LOS export, beating a general large model at that one job.
For most lenders the honest math points the other way. The GPU, the MLOps, and the patching are exactly the burden a partner should absorb, so you get the outcome, a cited and accurate answer on a loan file, without standing up an ML platform to get it. Buy the outcome, not the infrastructure.