Inference & Serving

Serving runtimes, Triton, vLLM, ONNX Runtime, and scalable endpoints.

  • 3 Tracked terms
  • Last 30 days Feed window
Search within this topic Ask AI about this topic All topics

What this topic collects on

An article joins this feed when it matches these terms. Each one is also a search of its own.

Latest in Inference & Serving


dev.to > gde > gemma-4-on-a-tesla-t4-qat-weights-decode-179x-faster-than-bf16-2fi4

Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16

2+ day, 7+ hour ago   (1298+ words) The GPU is already there. A T4 attached to a Compute Engine VM needs no queued resource, no instance launch and no image, so this rig has no provisioning tools at all. Everything it ships is about the software on the…...


dev.to > gde > serving-gemma-4-on-an-amd-mi300x-what-199-an-hour-buys-52h9

Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys

3+ day, 7+ hour ago   (1104+ words) The card is a DigitalOcean GPU droplet reached through AMD Developer Cloud (devcloud.amd.com) — same v2 API, same droplet ids, token from the My AMD Team account. Creating and destroying it are console actions, deliberately: both are dollar-per-hour decisions and…...


dev.to > gde > 2b-gemma-4-deployment-with-cloud-run-nvidia-l4-mcp-sdk-2x-and-claude-code-4ml3

2B Gemma 4 Deployment with Cloud Run, NVIDIA L4, MCP SDK 2.x, and Claude Code

1+ week, 3+ day ago   (1508+ words) This article provides a step by step deployment guide for Gemma 4 E2B to a Cloud Run hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment with Claude Code. This project…...


docs.vllm.ai > projects > vime > en > stable > advanced > external-rollout-engines.html

External Rollout Engines Roadmap??? Vime

1+ week, 6+ day ago   (685+ words) An external rollout engine is a vLLM engine that is not launched by the vime training job. Another system deploys and owns the engine lifecycle; vime connects to those engines during training, registers a router, and syncs updated actor weights…...


phoronix.com > news > Rustls-0.23.44-Released

Rustls 0.23.44 Released With ML-DSA Certificates Enabled By Default

1+ week, 6+ day ago   (200+ words) Rustls as the modern TLS library implementation written in the Rust programming language is out with a new feature release. This morning's Rustls 0.23.44 release enables post-quantum secure ML-DSA certificates by default... - Categories Computers Display Drivers Graphics Cards Linux Gaming Memory…...


dev.to > aws-builders > three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-what-it-doesnt-2cin

Three Gemma 4 Deployments on One T4G for Under $3: What the Runtime Changes, and What It Doesn't

2+ week, 5+ day ago   (1818+ words) This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared…...


dev.to > gde > gemma-4-in-pure-jax-what-ports-from-tpu-to-gpu-and-what-doesnt-3m09

Gemma 4 in Pure JAX: What Ports from TPU to GPU, and What Doesn't

3+ week, 2+ day ago   (1003+ words) This article is about running a hand-written Gemma 4 port in pure JAX on three different accelerators, and about the two places the abstraction leaks. The code is here: This project aims to serve one Gemma 4 checkpoint from one JAX port…...


dev.to > aws-builders > pure-jax-on-g5g-serving-gemma-4-on-graviton-and-a-t4g-3glo

Pure JAX on G5g: Serving Gemma 4 on Graviton and a T4G

3+ week, 2+ day ago   (1363+ words) This article provides a step by step deployment guide for serving Google's Gemma 4 on an AWS EC2 G5g instance using pure JAX. The code is here: This project aims to serve a modern open model on the cheapest whole CUDA GPU AWS…...