Hugging Face Transformers 5.17.0 adds HYV4, VibeVoice and five more model families

HYV4 leads the model additions

The largest addition is Hy4-Preview, a 780-billion-parameter mixture-of-experts language model that activates 49 billion parameters for each token and supports a 1-million-token context window. Each mixture-of-experts layer contains 256 routed experts and one shared expert that is always active, with eight routed experts selected per token.

Its attention stack combines four mechanisms. Multi-head Latent Attention compresses keys and values into a low-rank representation before expanding them to one key-value pair per query head. DeepSeek Sparse Attention uses a lightweight indexer to choose a subset of keys; following the linked IndexShare design, only layers designated as full run the indexer, while shared layers reuse the most recent full layer’s selection.

HYV4 also employs gated MLA with learnable attention sinks. Each head has a sink logit that enters the softmax but contributes no value. Independent Hyper-Connections replace a conventional residual path with multiple parallel residual streams, collapsing and redistributing them around each sublayer.

A material limitation applies to the implementation: Transformers does not execute HYV4’s multi-token prediction layers. Released checkpoints retain the relevant weights so other runtimes can use them for speculative decoding, but Transformers ignores those weights while loading the model.

Multimodal and speech coverage expands

VibeVoice adds a framework for high-fidelity, long-form, multi-speaker synthesis. According to the release, it places next-token diffusion inside an LLM structure and targets conversational audio such as podcasts and audiobooks with multiple participants. The release also fixes a VibeVoice quantized-cache bug, while noting that flaky generation export tests are skipped.

NeoMME brings 260-million- and 800-million-parameter multilingual, multimodal-native encoders from H Company. The models process text tokens and raw image patches together in one bidirectional Transformer rather than depending on a separately pretrained vision tower or causal language model. NeoMME-Retriever is tuned for visual-document retrieval from text or page screenshots, producing both multi-vector representations for MeanMaxSim late interaction and mean-pooled representations for cosine-similarity search.

Fun-ASR-Nano is an 800-million-parameter end-to-end speech-recognition model from Alibaba DAMO Academy’s FunAudioLLM team. Hugging Face says it covers Chinese, English and Japanese, including seven Chinese dialects and 26 regional accents. It also supports customized hotwords for specialized vocabulary and emits punctuation without a separate punctuation model.

The NVIDIA Canary-1B-v2 integration covers multilingual automatic speech recognition and speech-to-text translation. Canary combines Parakeet’s Fast Conformer encoder with a Transformer decoder using fixed sinusoidal positions, encoder cross-attention and tied input-output embeddings. A processor-generated prompt selects the task: matching source and target languages requests transcription, while different languages request translation.

NeuCodec extends XCodec2 with finite scalar quantization and a single codebook, a design intended to simplify downstream speech-language modeling. Its specified operating point is 50 tokens per second at 16 bits per token, equal to 0.8 kbps. It accepts 16 kHz audio and produces 24 kHz output through an upsampling decoder; the release also describes its encoding as resistant to bit-level errors on noisy or unreliable channels.

KimiLinear and runtime changes

KimiLinear adds Moonshot AI’s hybrid linear-attention architecture. Kimi Delta Attention gives each key channel an independent forget gate, allowing recurrent state to decay per channel instead of per head. Most layers use KDA, but every fourth layer retains full attention based on DeepSeek-V3-style MLA; its feed-forward blocks use a mixture of experts with a shared expert.

The release standardizes two- and three-dimensional vision rotary embeddings through a centralized RoPE frequency-computation module. Custom vision models that implement attention-layer or model-specific RoPE grid interleaving must migrate to modeling_rope_utils.py, making this the principal compatibility issue in the update.

Generation changes remove an accelerator synchronization previously performed at every decoding step, reducing per-step overhead, and stop remote Hub files from being downloaded unconditionally during generation. Other corrections cover encoder-decoder auto-compile cache checks and consistent past_key_values naming in AfMoE.

Cache handling now accepts non-static implementations in VoxtralRealtime and raises an explicit error when paged attention is invoked without a cache instead of failing silently. Documentation was also clarified for continuous batching and the limitations of sliding-window models.

Kernel updates support nested FLA imports when only fla-core is installed and warn when a Hub kernel falls back to a slower reference PyTorch path. KernelConfig can register standalone functions such as RoPE and optionally avoid inheriting default mappings. Quantization fixes address FP8 embeddings for Qwen models, missing FP8 tensor-parallel layer overrides and needless MXFP4 weight dequantization on XPU devices; AutoRound also gains support for five-, six- and seven-bit configurations.

Additional fixes include respecting shift_labels in decoder-only language- and vision-language-model losses, adding offload support to gradient checkpointing, correcting transformers chat generation-flag parsing and preventing Windows memory exhaustion from safe_open memory mapping by using a pread backend.

Source: Hugging Face Transformers 5.17.0 release notes.

Definition. Transformers 5.17.0 is a Hugging Face release that expands native model support while updating generation, caching, kernels, quantization and vision rotary embeddings.

Key takeaways

FAQ

Sources