What this work contributes
Presents a differentially private prediction framework that generates synthetic text by aggregating clipped per-token logits from inference over private records, without fine-tuning the underlying language model. A blending operation combines private and public inference to improve utility in the evaluated in-context-learning tasks.
When to read this work
Read this when exploring how to use sensitive examples for language-model predictions or synthetic demonstrations with differential privacy, particularly when updating model weights is undesirable.
Topics: Differential privacy · Large language models · In-context learning · Private text generation
Abstract
Large language models (LLMs) have significantly transformed natural language understanding and generation, but they raise privacy concerns due to potential exposure of sensitive information. Studies have highlighted the risk of information leakage, where adversaries can extract sensitive information embedded in the prompts. In this work, we introduce a novel private prediction framework for generating high-quality synthetic text with strong privacy guarantees. Our approach leverages the Differential Privacy (DP) framework to ensure worst-case theoretical bounds on information leakage without requiring any fine-tuning of the underlying models. The proposed method performs inference on private records and aggregates the resulting per-token output distributions. This enables the generation of longer and coherent synthetic text while maintaining privacy guarantees. Additionally, we propose a simple blending operation that combines private and public inference to further enhance utility. Empirical evaluations demonstrate that our approach outperforms previous state-of-the-art methods on in-context-learning (ICL) tasks, making it a promising direction for privacy-preserving text generation while maintaining high utility.
Tutorials and reading guides
- Privacy-preserving in-context learning: synthetic demonstrations and privacy budgets — How clipped logits, public context, and private token sampling can produce reusable ICL demonstrations, with careful accounting for privacy boundaries.
Cite this work
Download BibTeX · Download RIS
@inproceedings{bhusal2026aaai,
title = {Privacy Preserving In-Context-Learning Framework for Large Language Models},
author = {Bhusal, Bishnu and Acharya, Manoj and Kaur, Ramneet and Samplawski, Colin and Roy, Anirban and Cobb, Adam D. and Chadha, Rohit and Jha, Susmit},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {40},
number = {42},
pages = {35303--35312},
year = {2026},
month = mar,
doi = {10.1609/aaai.v40i42.40838},
url = {https://doi.org/10.1609/aaai.v40i42.40838},
eprint = {2509.13625},
archiveprefix = {arXiv}
}