Prompt Filtering: Developing a practical defense system against LLM jailbreaks

Abstract

Large Language Models (LLMs) have become central to modern artificial intelligence (AI) applications due to their remarkable ability to generate coherent, context-aware text. However, this capability introduces erabilities, particularly jailbreak attacks that manipulate the model into producing harmful or unethical outputs.This project addresses the growing challenge of detecting jailbreak prompts before reaching the model, through the development of a prompt-level filtering system. We gradually collected multiple available sources containing both benign and jailbreak examples. These datasets were progressively used in a series of experiments, with their merging employed as a key idea to increase the number and diversity of prompts. Various embedding techniques, including FastText, DistilBERT, RoBERTa, and Longformer, were employed to represent input prompts at different semantic levels. Classification was handled using XGBoost, chosen for several advantages such as its scalability and fast training time.The system was evaluated using standard metrics such as accuracy, Area under the precision-recall curve (AUPRC), and Attack Success Rate (ASR). Results showed that using diverse training data and high-quality embeddings significantly improves detection performance and robustness. The final implementation, deployed as a web-based application, demonstrates how the four embedding models handle jailbreak prompt detection. This research offers a scalable, practical framework for enhancing LLM safety through early threat identification.