COMPUTE-EFFICIENT SCALING OF LARGE SEQUENTIAL RECOMMENDATION MODELS UNDER DYNAMIC USER BEHAVIOR
World Journal of Information Technology · 26.09.2023 · 1 (1) · 62-71
DOI: 10.61784/wjit4126 ↗Kurz gefasst
Fachartikel in der Zeitschrift World Journal of Information Technology, online erschienen am 26.09.2023. Verfasst von AoWei Shen. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Transformer-based sequential recommenders are increasingly scaled up and retrained as interaction logs grow, yet user behavior drifts between retraining cycles, and repeated from-scratch training of large models consumes compute that rarely yields proportional accuracy. This paper studies how model scale, update strategy, and training compute interact under a streaming protocol in which a model trained on earlier blocks of a chronologically ordered log serves the next block. On MovieLens-1M and Amazon Grocery, widening a SASRec-style model from 6.6K to 397K non-embedding parameters raises training compute by 4.4-13.3x relative to width 32 without a consistent accuracy gain, and models that are not refreshed lose 36% and 82% of their NDCG@10, respectively. Building on these observations, we propose SPROUT, a compute-aware update framework that warm-starts each update from a recency-aware replay stream, expands capacity with an exactly function-preserving width-doubling operator for pre-LayerNorm Transformers, and accepts growth only when a paired validation test shows a gain larger than one standard error. Over four update cycles, SPROUT stays within 1.3% of the best periodic-retraining accuracy on MovieLens-1M with 44% less cumulative compute than the retrained small model and 93% less than the retrained large model, and it achieves the highest accuracy on Amazon Grocery, 5.0% above the best retraining baseline, with 47% less compute than retraining the large model. Its growth-free configuration offers the most economical operating point, reaching 99.1% and 102.1% of the retraining accuracy with 33% and 51% of its compute.
Zitieren
APA
Shen, A. (2023). COMPUTE-EFFICIENT SCALING OF LARGE SEQUENTIAL RECOMMENDATION MODELS UNDER DYNAMIC USER BEHAVIOR. World Journal of Information Technology, 1(1), 62-71. https://doi.org/10.61784/wjit4126
Vancouver
Shen A. COMPUTE-EFFICIENT SCALING OF LARGE SEQUENTIAL RECOMMENDATION MODELS UNDER DYNAMIC USER BEHAVIOR. World Journal of Information Technology. 2023;1(1):62-71. doi:10.61784/wjit4126
BibTeX
@article{Shen2023,
author = {Shen, AoWei},
title = {COMPUTE-EFFICIENT SCALING OF LARGE SEQUENTIAL RECOMMENDATION MODELS UNDER DYNAMIC USER BEHAVIOR},
journal = {World Journal of Information Technology},
year = {2023},
volume = {1},
number = {1},
pages = {62--71},
doi = {10.61784/wjit4126}
}
EUVIMED