TapGuard: Temperature-Calibrated Execute–Confirm–Block Routing for LLM-Powered Mobile Agents
DOI:
https://doi.org/10.51137/wrp.ijmat.1051Keywords:
Mobile Agents, Large Language Models, Prompt Injection, Safety Routing, Selective PredictionAbstract
Large language model-powered mobile agents can compose messages, manage files, operate financial applications, and act on content from other applications. Binary allow-or-refuse safeguards fit this setting poorly: some requests are safe to execute, some require confirmation, and some should be blocked. TapGuard is a task-level pre-execution router combining a compact semantic encoder, temperature-scaled three-class probabilities, a fixed rule floor, and confidence-triggered confirmation. Experiments used 150 canonical MobileSafetyBench release 3 tasks and 100 paired action-hint variants under instruction-template and task-category holdout protocols. Under template holdout, TapGuard lowered severity-weighted direct-execution exposure from 35.7% for the MiniLM semantic classifier to 27.7%, while low-risk direct execution declined from 64.0% to 54.0%. The frozen-cue router achieved still lower exposure but directly executed only 18.0% of low-risk tasks. TapGuard increased severe-risk blocking relative to MiniLM without achieving the highest three-way macro-F1. Category holdout showed the same broad trade-off. Privacy, offensiveness, and image-linked cases remained difficult, and generalization to unseen injection patterns was not established. These findings support evaluating mobile safety routing through both protection and preserved automation.
References
Ai, Q., Bu, P., Cao, Y., Wang, Y., Gu, J., Xing, J., Zhu, Z., Jiang, W., Zheng, Z., Song, J., & Jiang, Y. (2026). InquireMobile: Teaching VLM-based mobile agent to request human assistance via reinforcement fine-tuning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 32241–32255). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.acl-long.1487
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Article 3, pp. 1–13). Association for Computing Machinery. https://doi.org/10.1145/3290605.3300233
Debenedetti, E., Zhang, J., Balunovic, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems, 37, 82895–82920. https://doi.org/10.52202/079017-2636
Desai, S., & Durrett, G. (2020). Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (pp. 295–302). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.21
El-Yaniv, R., & Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53), 1605–1641. https://jmlr.org/papers/v11/el-yaniv10a.html
Feng, Y., Du, J., Wang, Q., Ma, Z., Niu, Q., Matsuo, Y., Feng, L., & Yu, L. (2026). CORA: Conformal risk-controlled agents for safeguarded mobile GUI automation [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2604.09155
Geifman, Y., & El-Yaniv, R. (2019). SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning (Vol. 97, pp. 2151–2159). PMLR. https://proceedings.mlr.press/v97/geifman19a.html
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (pp. 79–90). Association for Computing Machinery. https://doi.org/10.1145/3605764.3623985
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330). PMLR. https://proceedings.mlr.press/v70/guo17a.html
Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., & Kıcıman, E. (2024). Defending against indirect prompt injection attacks with spotlighting [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2403.14720
Lee, J., Hahm, D., Choi, J. S., Knox, W. B., & Lee, K. (2026). MobileSafetyBench: Evaluating safety of autonomous agents in mobile device control. Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37565–37573. https://doi.org/10.1609/aaai.v40i44.41090
Lee, J., Lee, D., Choi, C., Im, Y., Wi, J., Heo, K., Oh, S., Lee, S., & Shin, I. (2025). VeriSafe Agent: Safeguarding mobile GUI agent via logic-based action verification. In Proceedings of the 31st Annual International Conference on Mobile Computing and Networking (pp. 817–831). Association for Computing Machinery. https://doi.org/10.1145/3680207.3765248
Lee, J., Min, T., An, M., Hahm, D., Lee, H., Kim, C., & Lee, K. (2024). Benchmarking mobile device control agents across diverse configurations [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2404.16660
Li, W., Bishop, W., Li, A., Rawles, C., Campbell-Ajala, F., Tyamagundlu, D., & Riva, O. (2024). On the effects of data scale on UI control agents. Advances in Neural Information Processing Systems, 37, 92130–92154. https://doi.org/10.52202/079017-2925
Lin, Z., Li, J., Pan, S., Shi, Y., Yao, Y., & Xu, D. (2026). Mind the third eye! Benchmarking privacy awareness in MLLM-powered smartphone agents. Proceedings of the AAAI Conference on Artificial Intelligence, 40(42), 35626–35634. https://doi.org/10.1609/aaai.v40i42.40874
Liu, G., Ye, J., Liu, J., Li, Y., Liu, W., Gao, P., Luan, J., & Liu, Y. (2025). Hijacking JARVIS: Benchmarking mobile GUI agents against unprivileged third parties. In Proceedings of the 2nd International Workshop on Edge and Mobile Foundation Models (pp. 12–18). Association for Computing Machinery. https://doi.org/10.1145/3737902.3768354
Naihin, S., Atkinson, D., Green, M., Hamadi, M., Swift, C., Schonholtz, D., Kalai, A. T., & Bau, D. (2023). Testing language model agents safely in the wild [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2311.10538
Rawles, C., Clinckemaillie, S., Chang, Y., Waltz, J., Lau, G., Fair, M., Li, A., Bishop, W. E., Li, W., Campbell-Ajala, F., Toyama, D. K., Berry, R. J., Tyamagundlu, D., Lillicrap, T. P., & Riva, O. (2025, April 24–28). AndroidWorld: A dynamic benchmarking environment for autonomous agents [Paper presentation]. Thirteenth International Conference on Learning Representations, Singapore. https://openreview.net/forum?id=il5yUQsrjC
Rawles, C., Li, A., Rodriguez, D., Riva, O., & Lillicrap, T. (2023). AndroidInTheWild: A large-scale dataset for Android device control. Advances in Neural Information Processing Systems, 36, 59708–59728. https://doi.org/10.52202/075280-2609
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410
Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., Dubois, Y., Maddison, C. J., & Hashimoto, T. (2024, May 7–11). Identifying the risks of LM agents with an LM-emulated sandbox [Paper presentation]. Twelfth International Conference on Learning Representations, Vienna, Austria. https://openreview.net/forum?id=GEcwtMk1uA
Sun, Q., Li, M., Liu, Z., Xie, Z., Xu, F., Yin, Z., Cheng, K., Li, Z., Ding, Z., Liu, Q., Wu, Z., Zhang, Z., Kao, B., & Kong, L. (2026). OS-Sentinel: Towards safety-enhanced mobile GUI agents via hybrid validation in realistic workflows. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9529–9553). Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.acl-long.431
Wang, J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., & Sang, J. (2024). Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2401.16158
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33, 5776–5788. https://proceedings.neurips.cc/paper/2020/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Xu, Y., Liu, X., Sun, X., Cheng, S., Yu, H., Lai, H., Zhang, S., Zhang, D., Tang, J., & Dong, Y. (2025). AndroidLab: Training and systematic benchmarking of Android autonomous agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2144–2166). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-long.107
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z. (2024). Ferret-UI: Grounded mobile UI understanding with multimodal LLMs. In Computer Vision – ECCV 2024 (pp. 240–255). Springer. https://doi.org/10.1007/978-3-031-73039-9_14
Yuan, T., He, Z., Dong, L., Wang, Y., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., Wang, R., & Liu, G. (2024). R-Judge: Benchmarking safety risk awareness for LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 1467–1490). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-emnlp.79
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024). InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 10471–10506). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.624
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2025). AppAgent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (Article 70, pp. 1–20). Association for Computing Machinery. https://doi.org/10.1145/3706598.3713600
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Qi Xin (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
