Questions to ask before buying an enterprise GPU server
Buying · AI Node Works engineering team · Published
The short answer: Before comparing GPU models, settle five things: the workloads and users the server must handle, the facility it goes into (rack space, power and cooling), the software stack and its compatibility, how it will be managed and secured, and who supports it after delivery. The GPU choice follows from those answers. A proposal that does not address all five is incomplete, whoever writes it.
1. Workloads and users
- Which workloads: serving models to users, batch processing, fine-tuning, rendering, or a mix?
- Which models now, and which are you likely to want in a year?
- How many people or applications use it at the same time at peak, and with how much context?
- Is it one shared service, or several teams that should be kept apart?
- What does success look like, and how will you measure it?
These answers set the GPU memory you need in total and per card. Simultaneous users drive memory as much as model size does.
2. Facility: space, power and cooling
- How many rack units, how deep, and what weight can the rack and floor take?
- What power is available at the rack, on how many separate feeds, and through what sockets?
- Is there a UPS, and how long must it carry the load?
- How much heat can the room remove, around the clock?
- Who can access the room for installation and later service?
Do the arithmetic early. NVIDIA lists the RTX PRO 6000 Blackwell Server Edition at up to 600 W, configurable. Four cards at that setting are 2.4 kW and eight are 4.8 kW, for the GPUs alone. Processors, memory, drives, fans and power-supply losses come on top, and almost all of it becomes heat the room must remove.
3. GPU choice and topology
- How much memory per GPU, and how much in total?
- Will one model span several GPUs? If so, how are the GPUs connected? The RTX PRO 6000 Blackwell Server Edition connects over PCIe 5.0 and does not support NVLink.
- Should a GPU be shared between teams? Multi-Instance GPU (MIG) splits a supported GPU into isolated instances with their own memory and compute; NVIDIA lists up to four 24 GB instances on this card.
- Is the chassis a configuration the maker has validated for these GPUs? NVIDIA runs an NVIDIA-Certified program for servers tested with its GPUs.
- What is the upgrade path: empty slots, power headroom, or a second server?
4. Software and compatibility
- Which operating system and version does your IT team support?
- Which frameworks, runtimes and versions must run, and are they supported on this GPU and driver?
- For AMD GPUs, is the card and operating system listed in AMD's ROCm compatibility matrix for the release you need?
- Will workloads run in containers or virtual machines, and do the features you need require separate software licences?
- Who keeps drivers and runtimes updated, and how are updates tested before they reach production?
5. Management and security
- Does the server have a management controller with a standard interface such as Redfish, so it can be monitored and power-cycled remotely, out of band?
- How are firmware updates delivered and applied?
- Is secure boot available and enabled? NVIDIA lists secure boot with root of trust for the Server Edition GPU.
- Which network segment does the server sit on, and who can reach the management interface?
- How are access, logs and alerts integrated with your existing tools?
6. Delivery, acceptance and support
- What burn-in testing happens before delivery, and will you see the results?
- What acceptance test proves the server does what was agreed?
- What documentation comes with it: configuration, serial numbers, firmware and software versions?
- What warranty applies to the system and to each component, and who handles claims?
- What support is included after handover, how is an issue reported, and what response is agreed?
The checklist on one page
| Area | Why it matters | A good answer |
|---|---|---|
| Workloads and users | Sets total and per-GPU memory | Named models, peak users and context, with growth |
| Facility | Power and heat limit what can be installed | Rack, feeds, UPS and cooling confirmed in writing |
| GPU and topology | Decides how models split and teams share | Card edition, interconnect and partitioning explained |
| Software | Incompatibility stops deployment | Versions checked against the GPU and driver |
| Management and security | Someone must run it safely, remotely | Out-of-band management, update process, network placement |
| Delivery and support | Problems surface after handover | Burn-in evidence, acceptance test, written terms |