In a word: Take a video/GIF on your computer (whether it was shot, downloaded, or generated by a local AI model),
Turn it into a “sequence frame sprite” that can be used directly in the game engine——The whole process is run on your own machine, not on the cloud.。
⚠️ Project nature description: This document belongs to
fs7744/meowcute
projectFor personal experimentation and research onlyis not a commercial product, provides no warranty, and does not constitute a production-ready solution.
The model weights, scripts and processes involved in this article are only for technical exploration and learning purposes; if used for actual projects or commercial use,
Please verify the licenses and authorizations of relevant models and materials yourself.
1. Understand three words first
Many people are confused when they hear “sequence frame sprite” for the first time. In fact, when you take it apart, you can see three things:
- Sprite Sheet: A large picture composed of many small pictures arranged in a grid. All the action frames of the characters in the game are stuffed into a PNG, and the engine “cuts” them out according to the coordinates for playback, saving memory and loading quickly.
- Sequence Frames: The small pictures above are in order – the first picture lifts the foot, the second picture takes a step… they are played in a loop in order, and it looks like they are moving. The essence is from childhoodFlip book/page turning animation。
- Local: Refers to the entire process not calling any cloud API. The video is on your hard drive, the calculation is on your CPU/GPU, and the finished product remains on your machine. The benefits are:No money, no Internet, no material leakage, run as many times as you want.。
Connect them:Local video → Split into a stack of frames → Pick out key frames → Put together a sprite image → Loop playback verification. That’s what this tool does.
2. Why should “video” be turned into “sprite”?
Video and sprites are actually two ways to store the same action:
| Video (mp4/gif) | Sprites (png + index) | |
|---|---|---|
| player | Player/Browser | game engine |
| Can I set the skeleton? | cannot | Able (with skeletal animation) |
| Can it be changed frame by frame? | Disaster | Easy (export each frame individually) |
| Enter Unity/Godot | Want to transfer | Eat directly |
So when you already have the “desired actions” (such as walking, swinging a sword, and waiting), but what you lack are “materials that can be used in the game”, this method comes in handy:
- Reuse existing actions: Screen recording, motion capture clips, and GIFs downloaded elsewhere, without having to re-draw each frame by hand.
- AI-generated actions also work: Use a locally running video model (such as HunyuanVideo, Wan) to generate an animation, and then convert it into a sprite diagram – creativity and preview are fast, and the official material can also be implemented.
- Pixel/retro style pocket: The assembled frames can then undergo pixel refinement and be directly used as 16-bit pixel materials.
3. What does the entire assembly line look like?
┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ ┌──────────┐
│ ① 选择视频 │ → │ ② 拆解帧 + 挑帧 │ → │ ③ 精灵图 │ → │ ④ 动画预览 │ → │ 导出成品 │
│ 视频 / GIF │ │ 两把筛子去重 │ │ 网格拼接+索引 │ │ 循环播放看效果 │ │ PNG/JSON │
└─────────────┘ └──────────────┘ └─────────────┘ └──────────────┘ └──────────┘
Each step corresponds to a canvas node, and parameters can be adjusted individually, or the entire process can be run with one click.
① Select video
Import mp4/mov/webm/gif, firstPlay it again in situMake sure the clips are correct – don’t wait until you’re done tearing it down to find out you’ve routed the wrong clip.
② Disassemble the frame + pick the frame
Split the video into a stack of frames according to the “frame rate limit (default 12fps)”, “maximum number of frames (default 120)” and “start and end time”.
Then two sieves will help you reduce 200 frames to about 20 frames:
- Frame skip step (step): Only 1 frame is left for every N frames. If the movement is slow, turn it up, if the movement is fast, turn it down.
- Proximity filtering (simPct): The one that looks “similar” to the previous one will be automatically discarded. A threshold value of 0 means no filtering; after removing the node, it will tell you the “median difference between the frames of this material”, and it is most troublesome to adjust according to it.
After siftingIn the end, it’s you who has the final say——Check/invert selection/leave only the filtered results one by one. Why not let the program automatically draw 8 cards and be done with it? Because “which postures constitute a complete cycle” is an aesthetic decision that only you can see.
③ Sprite map
The selected frames are assembled into a PNG according to the grid, and a copy is attached.Index JSON(Record coordinates + anchor points for each frame).
cols = 0automatically arrange into a nearly square grid (ceil(√帧数)), will not be stretched into an extra long strip.- You can set the frame spacing (leaving 2px transparent space for Unity’s automatic slicing) and margins.
- share zoom: All frames are scaled by the same ratio. The frame with the leg raised will not be inexplicably enlarged. The character size is constant throughout the loop.
- The anchor point (pivot) is centered at the bottom by default, making it easier to align the soles of the feet in the engine.
④ Animation preview
It loops on the canvas according to the frame rate you set (default 12fps). You can adjust the display magnification and step by frame to check whether the movements are smooth and whether the seams are correct.
4. The “magic” behind it (popular version)
Why can the approximation “drag the slider to get instant results”?
backend toevery frameCalculate a 16×16 grayscale “fingerprint” (256 bytes of base64). The front end gets this string of fingerprints and compares the difference between the two frames in real time in the browser.No need to go back to the backend at all. So you drag the threshold slider and the deduplication results are immediately refreshed.
Why is there a “white border” when cutting out a video with a white background?
In a video with a white background, the pixels on the edges of the characters are actually a mixed color of “character color + background white” (not translucent, but a mixed color). Ordinary cutouts are only judged by color distance, and this mixed color edge will be left as the background, which looks like an unclean white circle.
This tool usesMixing ratioCriterion:|边缘色 - 背景| / |角色内侧色 - 背景| < 0.85 If it is, it will be judged as overflow and be trimmed. In this way, no matter it is white or gray, the edges can be cleaned.
Why is Shared Zoom important?
If each frame is individually scaled to a fixed cell, the frame where the leg is raised will cause the character to be enlarged due to the large movement range; the frame where the leg is not raised will be smaller. The character will expand and contract during loop playback. For shared scaling, first calculate the “maximum bounding box of the character in all frames” and scale it uniformly, and the size will be stable.
5. Project experience: pitfalls we encountered when making this tool
The following are not textbook conclusions, they are drawn from real practice. Each one is accompanied by the phenomenon, root cause and actual measurement after repair, saving time for latecomers.
Pitfall 1: A white border is cut out from a video with a white background, and it cannot be removed cleanly no matter what.
- Phenomenon: For videos of people with a white background, a circle of white is always left on the edge after cutting out, which looks very dirty.
- Misdiagnosis first: I thought it was because the translucent feathering was not handled properly. Quantitative findings
hard=TrueWhen alpha is strictly 0/255,Translucent pixel ratio measured 0.0%——The white edge is not a problem of feathering. - true root cause: The RGB of the circle of pixels itself is “character color × α + white background × (1-α)”Mixed colorsnot translucent. The old criterion uses absolute color distance (threshold 50), but the distance from an edge pixel mixed with 40% white to a white background is about 200can never be judged. Changing whether it is “closer to the background or closer to the inside” does not work: the mixed color pixels with α≈0.5 are equidistant on both sides, and the more they need to be corrected, the harder it is to judge.
- Amend the law: use insteadMixing ratiocriterion
|边缘色 − 背景| / |角色内侧色 − 背景| < 0.85(Modify only when >15% of the background is mixed in), a set of thresholds takes over deep/light/color bottom. - Actual measurement(User material 640², 21 frames on white background): Decontamination pixels
1716 → 3994;Edge ring mean RGB[75,74,76] → [13,11,13](The white edge changes back to the stroke color), the proportion of the ring “< 40 from the white background"0.5% → 0.1%。
Pitfall 2: Approximation deduplication, “continuous stationary segments” cannot be eliminated
- Phenomenon: There are often 5 consecutive frames in the video that are almost motionless (starting/stopping), but the frame-by-frame comparison only loses 4 frames, and the entire section of stillness still occupies the position.
- root cause: The criterion is “compared to the previous frame”. Once the difference between a certain frame and the previous frame is very small, it will be deleted, but each frame in the static segment is “approximately equal to the previous frame”, so only a small amount is lost.
- Amend the law: The criterion is changed to “and the previous onekeep framesCompare”. The entire segment that does not move continuously will be compressed into 1 frame.
- By the way, an interactive lesson:filterOnly change the display, do not change automatically checked. Otherwise, the 12 frames you carefully selected will be replaced by sliding the slider – this is absolutely unacceptable, and you must click “Apply Filter Results” to adopt it.
Pitfall 3: Why can deduplication achieve “zero delay by dragging the slider”?
- practice: The backend gives theevery frameCalculating a 16×16 grayscale “fingerprint” (256 bytes base64), the total of 120 frames is only about 30KB. The front end gets the fingerprint and calculates the difference between the two frames in real time in the browser.No backend at all. So drag the threshold slider and the results refresh immediately.
- Why not use large models/don’t go back to the backend?: Deduplication is a high-frequency interaction (drag the slider once to count it once). Once you go online, there will be hundreds of milliseconds of lag, and the experience will be completely ruined. The fingerprint is small enough to distinguish “whether it is the same gesture”, and any larger fingerprint is just a waste.
Pit 4:cols=0 I almost stretched the sprite map into a super long strip of 2736×136
- Phenomenon: 21 frames, 128px cells, if one row is arranged, you will get
2736×136(aspect ratio 20:1). Some engines/sampling are unfriendly and weird to look at. - Amend the law：
cols=0The semantics are changed to “automatic square“——Get the number of columnsceil(√帧数)21 frames become 5×5656×656the same video memory but sampling friendly. To align horizontally (Godot hframes / Unity single row) just on the nodeExplicitly give the number of columnsthe UI provides two buttons “Auto Square/Horizontal Line”. Note: oldspritesheetThe nodes remain arranged in one row (that is one horizontal bar in each direction, the semantics are different).
Pit 5: The pixel prompt says “Align to grid lines”, and the model actually draws a grid.
- Phenomenon: I want the image to be pixel blocks, and the prompt word is “Strictly align the boundaries to the grid lines”. As a result, the model draws a line between each block.Grid lines/lattice borderslike an Excel table rather than a picture strip.
- Amend the law: Changed to “The boundary falls on the grid, butDo not draw the grid lines themselves”, and added that “the whole picture is a continuous picture, not a square”. The picture is clean immediately.
- lesson: The prompt “Don’t draw X” is much safer than “Align X”; the model will concretize abstract constraints into something you didn’t expect.
Pitfall 6: Long tasks get stuck as soon as they are tested. It turns out that the testing method is wrong.
- Phenomenon: I used the interface test tool to run “Frame Splitting + Full Frame Cutout”, and the progress stopped at “21 frames ready” and never moved again.
doneAtNever set. but directlyawaitRun the same function in 20 seconds and it’s done. - root cause: Test Client (TestClient)Create a new event loop for each requestthe background coroutine is discarded when the request ends. Second-level tasks are fine, but long tasks such as cutting out pictures/videos will be permanently stuck.
- rule: Test client available for second-level tasks;Long tasks (picture cutout, video generation) must be launched with real services and then verified(The project contains
tools/test_clip_api.py: Temporary data directory + random port, kill after running, no data pollution).
Pitfall 7: Video generation model, don’t let it be the only source of formal production
- Phenomenon: Use HunyuanVideo/Wan to directly generate a “walking video” and then split the frames. The result will flicker, the character will drift, and the time consuming will fluctuate (the retry mechanism will cause one task to run three times to generate the video).
- position: Video model is suitable forAction Exploration/Previewis not suitable as the only production method for official game sprites. For the same action, the formal production process should be “Character Reference Picture → Standard 2D Skeleton → Deterministic Rendering of Sprite Frames”. The action can be applied to multiple characters, is editable, and does not drift.
- Division of labor: This tool locates the video as the “action source” – whether it is shot, recorded, or generated by AI. It is converted into a sprite image and then processed by the engine or pixel refinement. This division of labor is the most stable.
6. Joshosho
- Don’t select repeating poses at the beginning and end of the loop animation: The last frame is too similar to the first frame, and the playback will be “stuck”.
- Broadcast first and then dismantle: Make sure the source clip is correct before proceeding, so as not to find the wrong clip after disassembling it.
- Trust your own eyes to make the final cut: The filter only helps you reduce the burden. It’s up to you to decide which frames to choose.
- To make a transparent base material: u2net image cutout is turned on by default; for videos with white background, remember to pay attention to the “white border” processing above, which has been done for you.
- Want to enter Unity / Godot: Leave 2px frame spacing + use the exported index JSON, the engine will automatically slice it in the most trouble-free way.
- walking in multiple directions: This chain is “There is already a video → Convert to sprite image”; to make eight directions, you can use the “walking video/sequence frame/sprite whole image” node in the canvas to build it yourself.
7. Its relationship with “AI generates formal materials”
Video generation models are suitable forAction explorationmay not be suitable to be an official game elf.the only way to produce
(There will be flickering, character drift, and time-consuming fluctuations). This tool locates the video as an “action source”.
Convert to sprites before entering the engine or pixel refinement – this division of labor is more stable.
If you want the official production link of “character reference image → standard skeleton → deterministic rendering sprite frame”,
See the technical review document in the same directory Meowcute 视频生成序列帧方案分析.md。
8. One sentence summary
Convert local video to sequence frame sprite = convert your existing (or local AI generated) action video,
Offline frame splitting, frame picking, and jigsaw puzzles are turned into sprites and index files that can be directly consumed by the game.
No Internet connection, no money, no material exported from the machine, it’s always up to you to decide which frames to choose.
