LLM LATENCY OPTIMIZATION ON NON-STANDARD QUERIES USING HYBRID SEMANTIC CACHE MIDDLEWARE

Abstract

This research aims to optimize Large Language Model (LLM) latency in handling non-standard language queries through a Hybrid Semantic Cache Middleware architecture. The primary challenges in cloud LLM integration are high network latency and semantic matching failures on noisy text. The proposed method integrates the all-MiniLM-L6-v2 model with a static dictionary-based Text Normalization module and threshold optimization in FAISS. Experimental results show that the system successfully increased the Hit Rate from 11.00% to 58.00% under optimal conditions (Threshold 0.65). Furthermore, the average response latency was reduced from 4,494.00 ms (Cloud Baseline) to 17.98 ms, achieving a 99.6% speedup. This implementation proves that local caching with normalization effectively minimizes operational costs and network bottlenecks for Generative AI services in Indonesia.