All articlesThe Chronicle / Geodd

NVIDIA H100: What It Is, Key Specs, and What It Costs

The NVIDIA H100 is a data center GPU built on the Hopper architecture and released in 2022 to train and serve large AI models. It carries 80GB of high-bandwidth memory (94GB on the NVL version), fourth-generation Tensor Cores, and a Transformer Engine that runs transformer layers in FP8 precision. It comes in three forms (SXM, PCIe, and NVL). It is sold to businesses and cloud providers, not to gamers.

This guide covers why the H100 became the reference chip for AI work, how its specs differ by variant, and how to size one against a real model. It also covers what the H100 costs to buy or rent in 2026, with worked numbers you can reuse.

Geodd runs dedicated H100 capacity alongside a token-priced inference API.

Why the NVIDIA H100 matters for AI workloads

The H100 matters because it was the first mainstream GPU designed around low-precision transformer math, and it set the standard for large-model training and serving. NVIDIA claims up to 4X faster GPT-3 175B training and up to 30X faster inference on the largest models versus the A100. Those are projected, best-case figures. Teams more commonly report roughly 2X to 3X on real training jobs.

What the Hopper architecture changed

Three changes do most of the work. The Transformer Engine switches layers between FP8 and higher precision automatically, which cuts memory use and raises throughput without a manual accuracy fight. Fourth-generation NVLink gives 900GB/s of GPU-to-GPU bandwidth on the SXM version, so multi-GPU training scales better.

The third is memory. The SXM card pairs 80GB of HBM3 with 3.35TB/s of bandwidth. Hopper also adds Multi-Instance GPU (MIG), which splits one card into up to 7 isolated slices, and hardware-based confidential computing.

The H100 is a memory-bandwidth and low-precision machine, which is exactly what transformer workloads need.

Where the H100 earns its price

The H100 pays off in a few specific jobs:

  • Training or continued pretraining of large models across multiple GPUs
  • Fine-tuning models in the 30B to 70B range
  • Production inference for large language models where latency and throughput both matter
  • High-performance computing that needs fast FP64 and large memory

It is a poor fit for small models. If your weights fit in 24GB to 32GB, a consumer card delivers far more compute per dollar. It is also not a gaming card. It is built for data center racks and has no display outputs to speak of, and its price is many times that of a flagship gaming GPU.

NVIDIA H100 specs by variant

The H100 comes in SXM, PCIe, and NVL variants, and the differences in memory type, bandwidth, power, and interconnect matter more than the shared Hopper label suggests. Here are the figures side by side.

SpecH100 SXMH100 PCIeH100 NVL
GPU memory80GB HBM380GB HBM2e94GB HBM3
Memory bandwidth3.35TB/s~2TB/s3.9TB/s
FP8 Tensor Core (with sparsity)3,958 TFLOPS3,026 TFLOPS3,341 TFLOPS
Max power (TDP)Up to 700W350W350-400W
MIG instancesUp to 7Up to 7Up to 7
InterconnectNVLink 900GB/s, PCIe Gen5PCIe Gen5 128GB/sNVLink 600GB/s, PCIe Gen5
Form factorSXM modulePCIe dual-slotPCIe dual-slot, air-cooled

How to read the numbers

The headline TFLOPS figures assume structured sparsity, which only some workloads can use. Dense throughput is half the listed figure, so SXM FP8 is about 1,979 TFLOPS without sparsity. Use the dense number when you compare against other cards or your own benchmarks.

For inference, bandwidth often matters more than raw compute. Token generation reads the model weights from memory on every step, so the 3.35TB/s of the SXM card is the spec that most directly sets decode speed at large batch sizes.

SXM, PCIe, and NVL in practice

SXM modules sit on 4-GPU or 8-GPU baseboards inside HGX and DGX H100 systems. They draw up to 700W each and need a compatible server. PCIe cards fit standard servers at 350W, so they are far easier to deploy, but they have slower memory and no 900GB/s NVLink fabric.

Three H100 form factors side by side: an SXM module, a PCIe card, and a bridged NVL pair.

The NVL is a PCIe design with more memory per card. A bridged pair gives 188GB, which NVIDIA positions for models up to 70B parameters, such as Llama 2 70B. It cites up to 5X the performance of A100 systems on that model.

Pick the variant by memory and interconnect, not by the shared H100 name.

How to choose and size an H100 for your model

Start with your model's memory footprint, then pick the variant that holds it with room to spare. Most bad H100 purchases come from skipping this step. Teams buy by FLOPS and then discover the weights plus cache do not fit on one card.

Size memory before you pick a variant

Weights need roughly parameters multiplied by bytes per parameter. FP16 uses 2 bytes, FP8 uses 1, and 4-bit quantization uses about 0.5. The table shows weights only.

Model sizeFP16FP84-bit
8B16GB8GB4GB
32B64GB32GB16GB
70B140GB70GB35GB

Take a 70B model. In FP16 it needs 140GB, so it spans two 80GB cards. In FP8 the weights take 70GB of an 80GB H100, which leaves only about 10GB for the KV cache. That cache grows with context length and concurrent requests, so a single card handles short contexts and few users at best. Two GPUs, or an NVL pair with 188GB, give far more headroom.

Pick the variant

Work through these checks in order:

A four-step process listing checks for choosing an H100 variant, from single GPU fit to hosted API.

  1. If the weights fit in about 60GB to 70GB and you run one GPU, a PCIe or single SXM card is enough.
  2. If you train across several GPUs, choose SXM, because NVLink at 900GB/s keeps scaling efficient.
  3. If you serve a 70B-class model and need cache headroom, use an NVL pair or two GPUs.
  4. If traffic is low or bursty, skip the GPU entirely. Calling a hosted model through a token-priced API avoids paying for idle hardware.

Size the weights first, then add cache headroom, and only then compare variants.

H100 vs A100 and H200

The A100 has no FP8 support, and the H200 is the same Hopper compute with much more memory. The H100 sits between them.

A100 80GBH100 SXMH200
Memory80GB HBM2e80GB HBM3141GB HBM3e
Bandwidth~2TB/s3.35TB/s~4.8TB/s

Choose the H200 when your model or context length needs more than 80GB. Choose the A100 when FP8 throughput is not needed and price matters most. For everything in between, the H100 is the default.

What an NVIDIA H100 costs to buy or rent

A standalone H100 PCIe card sells for roughly $25,000 to $30,000, an 8-GPU DGX H100 system runs about $250,000 to $400,000, and cloud rental runs a few dollars per GPU hour. Street prices move with supply, so treat these as ranges.

Buying: card, server, and used

SXM modules are usually sold only inside complete systems, so the card price is only part of the bill. You also pay for the server, power, cooling, networking, and the people who run it. Used prices peaked at $80,000 to $120,000 per card in 2023 and have since eased, but they still tend to sit above list price.

A boxed H100 PCIe card in front of an eight-GPU server chassis with a power cable and fan module.

A few listings advertise an "H100 96GB." NVIDIA's own datasheet lists 94GB for the NVL, so verify the exact part number before you pay.

Renting: hourly rates and a worked break-even

Published on-demand rates for H100 instances in 2026 run from about $2.89 per hour for PCIe to about $4.39 per hour for SXM. Now work an example. One PCIe GPU running nonstop for a 720-hour month costs 720 × $2.89, about $2,081. At 8 hours a day it costs about $694.

Against a $25,000 card, nonstop rental takes about 12 months to break even. Against a $30,000 card it takes about 14.4 months. That is before you add the server, power, and cooling for owned hardware, so renting wins for most teams unless utilization stays near 100% for years.

Geodd's dedicated GPU access is bare metal and single tenant, billed hourly with SSH access. Ready in 60 seconds to 5 minutes, depending on the GPU type. Standard H100 80GB machines come in two sizes:

  • 1x H100: 20 vCPUs, 128GiB memory, 1,200GB storage
  • 2x H100: 40 vCPUs, 256GiB memory, 2,500GB storage

Export rules and availability

The H100 is not banned for general sale, but US export controls restrict shipping advanced AI chips to China and some other destinations. If you plan to move hardware across borders, check the current Commerce Department rules first. Lead times for owned hardware through authorized channels have been unpredictable, which is another point in favor of renting.

Rent until your GPUs run near-continuously for a year or more, then price the full cost of ownership.

Bottom line on the H100

The H100 is a Hopper-generation data center GPU whose strengths are fast memory, FP8 transformer math, and NVLink scaling. Choose SXM for multi-GPU training, PCIe for simple deployment, and NVL when a 70B-class model needs the extra memory. Size the weights and cache first, since that decides how many cards you need.

Buying costs about $25,000 to $30,000 per PCIe card before the server around it, and renting at a few dollars per hour is the better fit for most teams. To try one without the capital outlay, you can rent NVIDIA H100 80GB GPUs from Geodd, from single GPUs up to dedicated clusters of 128 to 500 GPUs.

The Chronicle / Bartosz Neuman
Build with Deploypad

Your next idea. A model API away.

Start building with inference and model APIs on Deploypad.

$10 USD Discovery credit

Explore inference and model APIs with $10 in Discovery credit. Enroll in Deploypad and unlock it with your first qualifying automatic top-up. Eligibility and offer details are shown in Deploypad.

Get started on Deploypad
Keep reading

More from Geodd.

All articles