# Heretic (AI) Heretic is an open source command-line tool that removes refusal behavior from [[Transformers|transformer-based]] [[Large Language Models (LLMs)]]. Its README calls that behavior "censorship (aka 'safety alignment')". You give it a model, and it gives you back a copy that answers most of the prompts the original refused. There's no training dataset to prepare, no [[AI Fine-Tuning|fine-tuning]], and you don't need to understand transformer internals. On an RTX 3090, a 4B model takes about 20 to 30 minutes. It was written by Philipp Emanuel Weidmann (p-e-w) and is licensed under the [[Affero General Public License (AGPL)|AGPL-3.0]]. ## How it works Heretic is built on directional ablation, better known as "abliteration". The idea comes from Arditi et al. (2024). Across 13 open chat models of up to 72B parameters, they found that refusal is driven by a single direction in the model's internal activations (the residual stream). Erase that direction and the model stops refusing. Add it, and the model refuses even harmless requests. Heretic computes that "refusal direction" for each layer: the difference between the average activations on a set of "harmful" prompts and a set of "harmless" ones. It then edits the weight matrices of two components (the attention output projection and the MLP down-projection) so they can no longer write along that direction. The edit lives in the weights, so what you get at the end is a normal model checkpoint. Abliteration has knobs: which direction to remove, how hard to ablate each layer, how that strength changes across layers. People who publish uncensored models, like Maxime Labonne (mlabonne) or huihui-ai, have been setting those by hand, one model at a time. Heretic automates it and hands the search to Optuna, an optimization framework, using its TPE (Tree-structured Parzen Estimator) sampler. Every trial gets two scores: the number of refusals left, and how far the model drifts from the original on harmless prompts. That drift is measured as KL divergence, a number for how different two probability distributions are (0 means the model behaves exactly like the original). The optimizer pushes both down at the same time. The README credits three design choices for the improvement over earlier tools: - The ablation strength follows a flexible curve over the layers instead of a constant value (Labonne had already experimented with non-constant weights) - The direction index is a float. For values between two layers, Heretic interpolates their directions, so the optimizer can try directions that no single layer produced - Attention and MLP get separate parameters. Weidmann found that MLP edits damage the model more than attention edits, so they can be applied more gently ## Why the KL divergence matters Earlier abliterations removed refusals but also degraded the model on normal tasks, and KL divergence is how that damage gets measured. The README compares versions of Google's [[Gemma|Gemma 3 12B]]: | Model | Refusals on "harmful" prompts | KL divergence on "harmless" prompts | |---|---|---| | Original | 97/100 | 0 | | mlabonne abliterated v2 (manual) | 3/100 | 1.04 | | huihui-ai abliterated (manual) | 3/100 | 0.45 | | Heretic (automatic) | 3/100 | 0.16 | Heretic matches the hand-tuned versions on refusals with a much lower divergence, which means the model keeps more of what it could do before. Those are the author's numbers, though. An independent comparison by Richard J. Young (arXiv:2512.13655, December 2025) tested Heretic, DECCP, ErisForge and FailSpy's abliterator on 16 instruction-tuned models from 7B to 14B. Heretic's optimized approach landed anywhere between 0.043 and 1.646 in KL divergence depending on the model, and on the subset they benchmarked, the simpler single-pass tools preserved math skills better. Their main finding applies to every tool: mathematical reasoning suffers the most from abliteration, with GSM8K scores moving between +1.51 and -18.81 points. ## Supported models Most dense models, many multimodal ones, several [[AI Mixture of Experts (MoE)|mixture-of-experts]] architectures, and some hybrid models like Qwen3.5 (from the [[Qwen]] family). Pure state-space models aren't supported out of the box. Bigger models need a lot more VRAM and can take hours; 4-bit [[AI Quantization|quantization]] (bitsandbytes) brings the memory needs down. According to the README, the community has published more than 5,000 Heretic models on [[HuggingFace]]. The output keeps the original architecture, so it runs wherever the original model did (see [[Running AI Models Locally]]). ## Trade-offs Heretic removes the model's safety training, and it doesn't distinguish between refusals. The annoying ones go (Korben cites creative writing, reverse engineering and medical questions), and so do the ones that exist for good reasons. After that, nothing inside the model stops it from helping with a harmful request, and the responsibility for how it's used moves entirely to whoever runs it. If you put an abliterated model in front of users, you need your own [[AI Guardrails]] around it (input and output classifiers, permissions), because the model doesn't carry any anymore. Arditi et al. wrote that their findings "underscore the brittleness of current safety fine-tuning methods." Heretic makes that practical for anyone who can run a command-line program, and it matters most for [[AI Open Weight Models|open-weight models]]. Refusal behavior that a lab trains into released weights during [[AI Post-Training|post-training]] ([[Reinforcement Learning From Human Feedback (RLHF)|RLHF]], safety passes) can be stripped afterwards by anyone with a gaming GPU. For open weights, safety training sets the default behavior; it can't enforce anything once the file is downloaded. The uncensored variant of [[DeepSeek V4.1 Flash]], published within a day of the release, shows how fast that happens. Korben mentions that researchers are working on defenses that make models resistant to abliteration. Until those ship, I treat the [[AI Alignment|alignment]] of an open-weight model as a property of the exact checkpoint you run. There are legitimate uses: researchers probing a model's own failure modes, red-teaming, and ordinary requests that commercial models over-refuse. I have it on my list to test with an agent. [[Obliteratus]] is the closest alternative: another AGPL abliteration toolkit, with many intervention methods and analysis modules to pick from. Heretic automates those choices instead. ## References - Heretic repository (GitHub): https://github.com/p-e-w/heretic - Heretic website: https://heretic-project.org - Korben, "Heretic - Virer la censure d'une IA en une commande" (French, 26 May 2026): https://korben.info/heretic-decensure-modeles-ia.html - Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction" (arXiv:2406.11717): https://arxiv.org/abs/2406.11717 - Richard J. Young, "Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation" (arXiv:2512.13655): https://arxiv.org/abs/2512.13655 - Maxime Labonne's article on abliteration (Hugging Face): https://huggingface.co/blog/mlabonne/abliteration - Heretic models on Hugging Face: https://huggingface.co/models?other=heretic ## Related - [[Obliteratus]] - [[AI Alignment]] - [[AI Safety]] - [[AI Guardrails]] - [[AI Open Weight Models]] - [[AI Post-Training]] - [[AI Fine-Tuning]] - [[Responsible AI]] - [[DeepSeek V4.1 Flash]] - [[Running AI Models Locally]] - [[Large Language Models (LLMs)]]