Video2026
Pika Labs · USA
A Pika video model built on the open MiniMax H3 weights: it makes video with stereo sound at once in six steps and takes up to 12 reference materials (images, video, audio) that set the shot direction. Two adapters: FL2VA for text and the first and last frames, Ref2VA for reference materials. Locally it runs at 480p and 768p, while 1080p, 2K and 4K, audio stems and prompt enhancement are available only through the paid Pika API. It suits multi-shot stories, motion graphics and product films. The license inherits the MiniMax H3 terms and does not apply in the EU, the UK, South Korea and the USA, which has to be checked before use. Running it needs server-grade graphics cards.
- ads and product films with ready sound
- animation from reference images and video, multi-shot stories
- motion graphics and title sequences
- Sizes
- 33B (a fine-tune of MiniMax H3, adapters)
- Hardware
- from: Cluster
Video2026
Fudan University, Tencent Hunyuan and Zhejiang University · China
Creates video together with sound: from a first frame and a description you get a clip with speech, effects and music. Native generation at 720p, 1080p and 2K is stated. In the third-party FreeVideo build (1280 by 720, 8.5 seconds) it runs on one RTX 30, 40 or 50 series card with 12 GB or more, but it will not be fast.
- animating a picture into a clip with sound
- ad and training videos with voice-over
- prototypes of scenes for video and animation
- Sizes
- not stated on the card (based on MOVA-360p)
- Hardware
- from: 1 GPU
Music and sound2026
HeartMuLa Team · not disclosed
An open model for generating songs with vocals in Chinese, English, Japanese, Korean and Spanish, plus a codec and a lyrics transcription model.
- Songs and jingles from lyrics
- Music for videos
- Transcribing song lyrics
- Sizes
- 3B
- Hardware
- from: 1 GPU
Music and soundGGUF2022–2026
m-a-p (Multimodal Art Projection) · UK / China
A music encoder: turns a track into a numeric representation used to detect genre, mood, key and rhythm. MERT-v2 handles full songs up to 6 minutes.
- Automatic tagging of a music catalog
- Finding similar tracks
- Detecting genre, mood and tempo
- Sizes
- 95M – 632M
- Hardware
- from: Laptop
Acoustic monitoring2019–2026
Google · USA
A tiny model covering 521 sound events, including alarms, breaking glass and screams: it fits on a microcontroller and runs without a GPU. A microphone can capture people voices, which is personal data, so check the procedure with a lawyer.
- Fast sound labelling right on the device
- Detecting alarm sounds on a site
- Picking interesting fragments out of a continuous recording
- Sizes
- 3.7M
- Hardware
- from: Laptop
Music and soundGGUF2025–2026
M-A-P and HKUST · China
Generates full songs with vocals and accompaniment from lyrics and a style description: English, Chinese, Japanese, Korean.
- Songs and jingles from lyrics
- Demo versions of tracks
- Music for videos
- Sizes
- 0.5B – 7B
- Hardware
- from: 1 GPU
VideoGGUF2025–2026
Sand AI · China
Video generated chunk by chunk in sequence, so a clip can be extended indefinitely. MAGI-2 produces video with sound.
- Long videos with continuation
- Video with sound
- Animating images
- Sizes
- 4.5B – 114B-A6B
- Hardware
- from: 1 GPU
Voice: speakers and sound2022–2026
Community: Ultimate Vocal Remover (Anjok07), ZFTurbo, MVSep · International community
A large open collection of models for separating vocals from music and noise: MDX-Net, BS-RoFormer, Mel-RoFormer, SCNet. The quality leaders for vocals among open solutions.
- Clean vocals from a recording with music
- Backing tracks and stems for karaoke
- Removing background music and noise from videos
- Sizes
- from tens to hundreds of millions of parameters
- Hardware
- from: Laptop
Voice: speakers and sound2022–2026
Meta AI, then Alexandre Défossez · France
A classic model that splits a track into vocals, drums, bass and the rest. The v4 hybrid transformer version remains the benchmark; the project is now maintained by its author in his own repository.
- Separating vocals from music in a recording
- Backing tracks and karaoke stems
- Cleaning speech in videos with background music
- Sizes
- tens of millions of parameters
- Hardware
- from: Laptop
Video2024–2026
Lightricks · Israel
A fast video model; with LTX-2 it generates video with sound and speech in one go. Camera and pose control, lightweight versions available.
- Ad videos with sound
- Video from a product photo
- Voiced scenes for social media
- Sizes
- 2B – 22B
- Hardware
- from: 1 GPU
Music and soundRUGGUF2025–2026
ACE Studio and StepFun · China
Fast generation of songs with vocals in 19 languages, including Russian: a full song in seconds, editing of individual parts and style changes.
- Songs and jingles for ads
- Background music for videos
- Demo versions of tracks
- Sizes
- about 2B – 4B
- Hardware
- from: Laptop
Acoustic monitoring2019–2026
University of Surrey · UK
The classic set of convolutional networks that label sound across the 527 AudioSet categories, from machinery noise and alarms to breaking glass and screams. The system does not make a diagnosis, it gives you a reason to check the unit before it fails. A microphone can also capture people voices, which is personal data, so check the procedure with a lawyer.
- Labelling what is happening in a microphone recording
- Detecting alarms, breaking glass, screams
- A base for your own model tuned to one shop floor
- Sizes
- from 5.9M (CNN6) to 81.9M (CNN14)
- Hardware
- from: Laptop
Music and soundGGUF2024–2026
Stability AI · UK
Generates short music clips and sound effects from a description. Version 3 is split into separate models for music and for sounds.
- Sound effects for videos and games
- Background music and jingles
- Interface sounds
- Sizes
- about 0.5B – 2.3B
- Hardware
- from: Laptop
Acoustic monitoring2023–2026
Xiaomi · China
Compact sound-labelling models from 5.5M to 86M, with ONNX and INT8 builds that run on an ordinary CPU and on a board next to the equipment. The system does not make a diagnosis, it gives you a reason to check the unit before it fails.
- Labelling sound on modest hardware and on site
- Detecting alarm sounds and abnormal noise
- Fine-tuning for your own set of equipment sounds
- Sizes
- 5.5M - 86M
- Hardware
- from: Laptop
Acoustic monitoring2024–2026
Xiaomi · China
A general-purpose audio encoder trained on 272 thousand hours of speech, music and noise: it gives features on top of which you train your own classifier of abnormal sounds. The system does not make a diagnosis, it gives you a reason to check the unit before it fails.
- Audio features for your own anomaly model
- Finding similar fragments in a recording archive
- Fine-tuning for the sounds of one production line
- Sizes
- 86M - 1.2B
- Hardware
- from: Laptop
Avatars2026
SII-GAIR and Sand.ai · China
Generates video of a talking person with sound in one go: a single transformer processes text, video and audio. Speech in 7 languages; Russian is not among them. Fast distilled versions are available.
- Presenter video from a script
- Ad videos with a talking character
- Training videos with a narrator
- Sizes
- 15B
- Hardware
- from: 1 GPU
Music and sound2025–2026
Alibaba Tongyi (FunAudioLLM) · China
Generates and edits audio for video, text or audio, first "reasoning" about the scene with a multimodal model. PrismAudio is the next version for video-to-audio.
- Audio for video based on the scene
- Editing individual sounds in a track
- Sound effects from a description
- Sizes
- size not stated on the model card
- Hardware
- from: 1 GPU
VideoGGUF2026
OpenMOSS / MOSI · China
Generates video with sound in one pass: lip-synced speech, effects and ambience. A 32B-parameter MoE architecture, with 360p and 720p versions.
- Short clips with speech and sound from a description
- Ad scenes with dialogue
- Video prototypes for storyboards
- Sizes
- 32B-A18B
- Hardware
- from: 1 GPU
Computer vision2025
Meta · USA
Meta's family of encoders for images and video, and with PE-AV also for audio. PE-Core searches by text more accurately than SigLIP 2 (per Meta); small versions are available.
- Search photos and videos by description
- Catalog labeling and tagging
- Search across audio and video (PE-AV)
- Sizes
- size not stated on the model card
- Hardware
- from: Laptop
Voice: speakers and sound2025
Meta · USA
A model that cuts the sound you need out of a recording based on a text description, a mark on the video or a time range: a voice, an instrument, noise. Weights are available on request.
- Isolating one person's voice from a noisy recording
- Removing unwanted sound from a video
- Splitting a recording into separate sound sources
- Sizes
- small, base, large (5 to 15 GB of weights)
- Hardware
- from: 1 GPU
Acoustic monitoring2025
Carnegie Mellon University and co-authors (the ESPnet project) · USA
A fully open reproduction of BEATs: code, training recipes and weights are all published. Teams pick it when they need a transparent base for their own sound model. The system does not make a diagnosis, it gives you a reason to check the unit before it fails.
- A transparent base for your own audio model
- Fine-tuning on your own equipment recordings
- Labelling sound events
- Sizes
- base and large versions, exact sizes are listed on the model cards
- Hardware
- from: Laptop
Video2025
Character.AI · USA
Generates video together with sound and speech from text or an image: two branches (video based on Wan 2.2 and a 5B audio branch) run in sync. Needs 24–32 GB of GPU memory.
- Short clips with talking characters
- Animating an image with voice-over
- Ad scene prototypes
- Sizes
- 11B
- Hardware
- from: 1 GPU
Music and sound2025
ASLP-lab (Northwestern Polytechnical University) · China
Fast generation of a full song with vocals from lyrics and a style sample, up to several minutes long.
- Songs and jingles from lyrics
- Music for videos
- Demo versions of tracks
- Sizes
- about 1.1B
- Hardware
- from: Laptop
Music and sound2025
Tencent Hunyuan · China
Generates studio-quality (48 kHz) audio for video from the picture and a text prompt: footsteps, impacts, ambience, in sync with the action on screen.
- Foley and sound effects for video
- Sound for AI-generated ads
- Sound design for short videos
- Sizes
- not stated on the model card (weights about 10 GB; XL version with memory offloading)
- Hardware
- from: 1 GPU
Acoustic monitoring2024–2025
Shanghai Jiao Tong University and Peng Cheng Laboratory · China
A self-supervised sound understanding model that is markedly cheaper to train on your own data than its predecessors. The system does not make a diagnosis, it gives you a reason to check the unit before it fails.
- Training your own audio model on modest hardware
- Audio features for spotting abnormal operating modes
- Labelling sound events on a site
- Sizes
- 90M (base) and 309M (large)
- Hardware
- from: Laptop
Acoustic monitoring2024–2025
MIT CSAIL and MIT-IBM Watson AI Lab · USA
A sound understanding model built on state spaces instead of a transformer: it handles recordings hours long, which suits continuous listening to a production line. The system does not make a diagnosis, it gives you a reason to check the unit before it fails.
- Processing long continuous recordings
- Finding a rare event across a multi-hour shift
- Labelling sound events on a site
- Sizes
- 30M (small) and 49M (medium)
- Hardware
- from: Laptop
Music and sound2024
University of Illinois and Sony AI · USA / Japan
Adds sound to silent video: generates noises and sound effects in sync with the on-screen action, from the video and a text prompt. One of the first strong open Foley models.
- Sound effects for silent video
- Sound for clips from AI generators
- Draft sound design for editing
- Sizes
- size not stated on the model card
- Hardware
- from: Laptop
Music and sound2024
Tencent AI Lab · China
The MuQ music encoder and the MuQ-MuLan model, which matches music and text: you can search for tracks by a description in English or Chinese.
- Searching music by text description
- Tagging tracks by genre and mood
- Finding similar music
- Sizes
- 300M – 700M
- Hardware
- from: Laptop
Music and sound2023–2024
Meta · USA
Generates instrumental music from a text description or a sample melody. One of the first open models of its kind.
- Draft music sketches
- Music for video prototypes
- Research
- Sizes
- 300M – 3.3B
- Hardware
- from: Laptop
Music and soundNot maintained2023
LAION · Germany
CLIP for audio: maps audio and text into a shared space. Lets you search sounds and music by description and classify them without training. Text must be in English.
- Search sounds and music by description
- Automatic tags for an audio library
- Recognizing sound types (siren, breaking glass, voice)
- Sizes
- size not stated on the model card
- Hardware
- from: Laptop
Acoustic monitoringNot maintained2022
Microsoft · USA
A self-supervised sound understanding model that became the base for many current systems: teams take it as a starting point and fine-tune it on their own equipment sounds. The system does not make a diagnosis, it gives you a reason to check the unit before it fails.
- A base for your own sound understanding model
- Audio features for spotting abnormal operating modes
- Labelling sound events on a site
- Sizes
- a ViT-base class model, around 90M parameters
- Hardware
- from: Laptop
Music and soundNot maintained2022
MIT · USA
A classic 2021 sound recognition model: detects 527 AudioSet event classes (siren, barking, breaking glass, music). Lightweight, runs without a GPU, in Transformers since 2022.
- Sound event recognition
- Tagging an audio archive
- Detecting alarm sounds
- Sizes
- about 87M
- Hardware
- from: Laptop
