Best Medical LLM (2026): Open-Source Medical Models You Can Run on Your Own Servers

Medical language models fall into two groups. Some are fine-tuned specifically for medicine, such as Google’s MedGemma and Baichuan’s M-series. Others are general open-weight models that happen to score well on health questions. The general group improves every few months, so a medical fine-tune of an older base can lose its lead quickly.

This guide compares both groups as they stand on 22 October 2026. Sizes, licences, context windows and release dates come from each model’s Hugging Face card or vendor page, and serving support from the vLLM, llama.cpp and Ollama listings. Benchmark numbers are the vendors’ own unless we say otherwise. For general model choices by GPU size, see the local LLM guide organised by VRAM.

Quick picks for clinical and medical text

Medical fine-tunes versus general open-weight models

A medical fine-tune starts from a general base and adds training on clinical text, exam questions, records and, for multimodal models, medical images. The gain over the base can be large. Google reports that MedGemma 27B text scores 87.7 on MedQA against 74.9 for Gemma 3 27B, the model it was built from.

The catch is the base. MedGemma 27B is built on Gemma 3, released in 2025, while Gemma 4 arrived in 2026 with a 256K window. Gemma 4 31B reports 61.3% on the multimodal MedXpertQA set, but publishes no MedQA or HealthBench figures, so the two cannot be compared directly from the cards. Qwen3.6 and Qwen3.8 cards carry no medical benchmarks at all.

In practice, most teams run one of each. A medical model handles clinical vocabulary, abbreviations and structured extraction, and a general model handles long policy documents, tool calls and anything that needs the newest instruction following. A router can send each request to the right one.

The shortlist: medical LLM models checked in October 2026

Older medical fine-tunes built on Llama 2 or Llama 3, such as the Med42 line, still circulate on Hugging Face. Their bases are two or three generations behind the models above, so we left them out. For any medical model you find, read the licence before the benchmark table, because research releases often forbid commercial or clinical use.

What MedQA and HealthBench scores do and do not tell you

Most medical claims rest on a handful of tests:

• MedQA uses multiple-choice questions in the style of the US medical licensing exam. It measures recall and reasoning over a fixed set of options.
• MedMCQA and PubMedQA do the same for Indian entrance exams and for yes, no or maybe answers about research abstracts.
• MedXpertQA is a harder expert-level set, with a multimodal part that includes images.
• HealthBench, published by OpenAI in 2025, grades open-ended health conversations against rubrics written by physicians. A model does the grading.

None of these measures what a hospital usually needs: an accurate summary of a 40-page record, a coding suggestion that matches the documentation, or a patient letter at the right reading level. Public questions also leak into training data. The MedGemma 1.5 card itself advises developers to validate on data that is not publicly available, to reduce contamination risk.

There is also a gap between vendors. Baichuan’s and Tether’s HealthBench results are self-reported, and the QVAC post scores with judge models of its own choosing. Use benchmark tables to build a shortlist of two or three candidates, then decide with your own cases.

MedGemma on-premises: 4B, 27B and the 1.5 update

MedGemma is the family most teams start with, partly because it fits ordinary hardware and partly because Google publishes detailed cards. There are three checkpoints to know.

MedGemma 27B text is the strongest at clinical question answering in the family. Google reports 87.7 on MedQA with zero-shot prompting and 89.8 with best-of-five sampling, plus 84.0 on AfriMed-QA. It accepts at least 128K tokens of input and writes up to 8,192.

MedGemma 27B multimodal adds a SigLIP image encoder that turns each 896 by 896 image into 256 tokens. It scores slightly lower on text questions, 85.3 on MedQA, but reads chest X-rays and answers questions over synthetic FHIR records, where Google reports 90.5 on its EHRQA set.

MedGemma 1.5 4B, released on 13 January 2026, is the only 1.5 checkpoint. Google added support for CT and MRI volumes, whole-slide pathology, comparison of chest X-rays over time, anatomy localisation and extraction of values from lab reports. Google’s announcement reports lab report extraction rising from 60% to 78% macro F1 and EHR question answering from 68% to 90% against the first 4B model. Google released MedASR, a speech-to-text model for medical dictation, on the same day.

To run MedGemma with Ollama, pull medgemma:27b for the multimodal 27B or medgemma1.5:4b for the update. For the text-only 27B, the model card shows vllm serve and SGLang commands. All three are gated on Hugging Face until you accept the terms.

GPU memory for medical models

Weights dominate memory, so the table lists the file size of each build. Add room for context: a long discharge summary plus a retrieved guideline can reach 20,000 tokens, and the KV cache for that grows with model depth. Our VRAM calculator estimates both parts for a given model and context length.

A practical reading of that table: an 8 GB card runs MedGemma 1.5 4B, a 16 GB card runs gpt-oss-20b, a 24 GB card runs MedGemma 27B at 4-bit, an 80 GB card runs gpt-oss-120b, and Baichuan-M3 needs a multi-GPU server. Baichuan’s own speculative decoding example uses eight 96 GB H20 cards.

Running medical models with vLLM, llama.cpp and Ollama

Ollama is the quickest way to try a model on a workstation. For a shared service with many clinicians, vLLM or SGLang gives better throughput and an OpenAI-compatible API that other systems can call.

Licences: Health AI Developer Foundations terms and Apache 2.0

MedGemma is not under Apache 2.0. Its weights come under Google’s Health AI Developer Foundations (HAI-DEF) terms, and Google says the HAI-DEF models are free for research and commercial use. The terms add duties that matter in a hospital or a health-tech product:

• you are responsible for deciding whether a use of the model or its outputs is appropriate, and Google provides no medical advice;
• uses that could lead a regulator to treat Google as a medical device manufacturer are prohibited, along with uses that break the law;
• if you distribute the model or a product built on it, including as a hosted service, you pass on the use restrictions and seek health regulatory authorization where it applies;
• you indemnify Google for claims arising from your use.

Baichuan-M2, Baichuan-M3, II-Medical-8B, gpt-oss, Gemma 4 and Qwen3.6 are under Apache 2.0. Two caveats apply. II-Medical’s card says the model is not suitable for medical use, and Tether describes MedPsy’s licence as applying to research and educational purposes, so neither belongs in a production clinical workflow without legal review.

Matching a model to the clinical task

Clinical note summarisation. MedGemma 27B text is the first choice, because it handles clinical abbreviations and long inputs. Summaries must state only what the note records, and the clinician checks the result against the source before signing. Our clinical documentation support use case shows how that review step fits into the workflow.

Medical coding and billing. Code suggestions need exact matching against ICD-10-CM, CPT and payer rules, which no model memorises reliably. Use retrieval over the current code sets and your payer policies, a general model with strong instruction following such as gpt-oss-120b or Qwen3.6, and a coder who approves each suggestion. The medical coding validation use case describes the checks.

Literature review. Searching and summarising papers needs long context more than medical tuning. Gemma 4 31B and Qwen3.6-35B-A3B take 256K and 262K tokens, and retrieval with citations keeps each claim tied to a source. See the research and literature review use case.

Patient letters and instructions. The hard part is plain language at the right reading level, not medical knowledge. A general model writes clearer letters; a medical model can check the clinical content. A clinician or nurse approves before anything is sent, as in our patient communication use case.

Imaging-adjacent text. MedGemma 1.5 4B and the 27B multimodal model can read images alongside reports, for tasks such as drafting a structured finding list or comparing a new chest X-ray report with a prior one. These are drafting aids for radiologists. Neither model is cleared to make a diagnosis.

Evaluating a medical LLM on your own cases

Run a short, documented test before any model reaches clinicians:

1. Pick 50 to 200 real cases per task, de-identified, covering common and rare presentations and your specialty mix.
2. Write the grading rubric first, with clinicians: factual accuracy against the source, omissions, invented findings, tone and format.
3. Compare two or three models on identical prompts, with the same retrieval and the same context length you will use in production.
4. Record the model version and quantization. A 4-bit build can behave differently from the BF16 weights a vendor benchmarked.
5. Repeat the test after every model update and keep the results, so a later reviewer can see why a model was approved.

How VDF AI fits clinical model selection

VDF AI is a self-hosted platform for private assistants, retrieval over your own documents and governed agents, installed on your servers, in a private cloud or air-gapped. Patient records, retrieval indexes, conversation history and audit logs stay in your environment, and models such as MedGemma or gpt-oss run on GPUs you control.

VDF AI Router registers local model endpoints served by Ollama, vLLM or a custom server, so a medical model can take clinical extraction while a general model takes long policy documents. Air-gap mode keeps every request on local models, and pinned model versions mean a model does not change under a validated workflow. Agents can stop for human approval at each step that matters, and every prompt, retrieved source and output lands in the audit log.

The clinical documentation agent structures notes from what was recorded and leaves the signature with the clinician. For the wider picture, see the healthcare and life sciences solution and the private GPT guide for healthcare. Hospitals that want to test models on their own hardware first can begin with a three-month trial of the on-premises platform.