MS Thesis Defense: Thang Nguyen

"Toward General-Purpose LLMs: From domain alignment to multimodal and multi-agent systems"

Abstract

Large language models (LLMs) have shown remarkable capabilities, but their generalization across domains, modalities, and collaborative settings remains limited. This work explores how LLMs can be adapted along three dimensions: domain specialization, multimodal processing, and multi-agent reasoning. First, we introduce a reward-guided retrieval mechanism that fine-tunes the retrieval component of a language model using preference-based supervision, improving response relevance in specialized domains. Building on this, we design a multi-agent framework for complex information-seeking tasks, where distinct agents handle query clarification, evidence extraction, and answer synthesis, enabling robust reasoning without additional model training. Extending beyond language, we investigate how LLMs can operate in visual domains. We adapt pretrained models for image restoration by tokenizing images and applying fine-tuning techniques. This enables flexible restoration across degradation types and supports multi-agent collaboration through an extension that structures the restoration process into interactive components.

Finally, we present an exploratory study of LLMs in cybersecurity. In a simulated enterprise network, an LLM-based agent performs monitoring, analysis, and deception to defend against attacks. While not optimized for state-of-the-art performance, the study reveals practical insights into the potential of LLMs for autonomous decision-making in real world environments. Together, these studies demonstrate how LLMs can be reconfigured to address a broad range of tasks, offering a deeper understanding of their versatility across domain-specific reasoning, multimodal processing, and collaborative agent systems.

Thesis Committee

• Professor Peter Chin (Chair)
• Professor Yu-Wing Tai
• Professor Wesley Marrero

Contact

For more information, contact Thayer Registrar at thayer.registrar@dartmouth.edu.