Anonymous submission · AAAI 2027 · under double-blind review

DigitCode: Symbolic Tokenization of Hand Motion by Anatomical Units

Anonymous authors — identities withheld for review.

The paper PDF and the HandTok testbed will be linked here once the review period ends.

Abstract

Hand motion carries the finest-grained information in human activity, yet the representations behind hand generation, understanding, and robot learning are overwhelmingly continuous—joint angles or MANO parameters. These are accurate but unstructured: a finger cannot be indexed or edited as a symbol, and nothing marks a pose as anatomically valid. Discrete symbolic representations supply exactly this structure, and Hand Labanotation (HL) has shown they are feasible for the hand, writing motion as a T×40 grid of one fixed direction symbol per bone. Building on this grid, we ask the question underneath it: the anatomical unit a symbol should span—bone, finger, or whole hand. DigitCode answers it by adapting, grouping, and layering HL's alphabet along the hand's unit hierarchy within one code, cutting the symbolic representation's quantization error by three quarters. The lever is the unit, not the quantizer family: at a fixed unit, training-free and learned strong quantizers are interchangeable on reconstruction, while moving down the anatomical hierarchy is what shifts accuracy. The hierarchy also tracks what downstream tasks need. Because a finger is a genuine, enumerable unit, one per-finger token doubles as a training-free, editable handle for jobs a continuous representation cannot address—repairing malformed generated hands, and retargeting them onto robots. We release HandTok, a reproducible testbed, so hand tokenizers can be compared unit-for-unit.

14.71° → 3.26°

held-out reconstruction error at HL's own rate (4.75 vs. 4.70 bits); 1.86° at 6.75

5 tokens

one per finger — enumerable, legal by construction, locally editable

0 training

the codebook is k‑means; detection and repair need no learned model

6 tasks

the useful unit tracks the task across bone, finger, and whole hand

The question

Every discrete representation decides two things: how finely to quantize, and what a token should span. Tokenization research has overwhelmingly pursued the first—larger codebooks, learned and residual quantizers—while the second is usually inherited from the data's grid and left unexamined. For signals with real compositional structure, we argue the second decision is the one that matters, and hand motion is an unusually clean place to test it: the hand arrives with its own unit hierarchy, bone → finger → whole hand.

Hand Labanotation writes a hand as one direction symbol per bone, so a two-hand sequence becomes a T×40 grid of discrete symbols over uniform time steps—in effect, the pianoroll of hand motion. We keep that grid and change only the unit a symbol spans.

A two-hand motion sequence written as a T by 40 grid of discrete direction symbols, five finger blocks per hand, colour-coded by the direction each symbol names. — **Hand motion as a symbolic grid.** Each of the 40 bones (20 per hand) becomes one of 26 direction symbols per frame; colour encodes the *direction* a symbol names (lightness falls with the angle from “forward,” hue is the azimuth—the polar key at right places all 26), so a small pose change reads as a small colour shift. Blocks are fingers, so the vertical axis is already an anatomical hierarchy—the paper's question is which level of it a single token should occupy.

Method — three changes, each driven by a property of hand motion

01 · fit

Bone directions are anisotropic

68.1% of all bone directions fall on a single face of HL‑26's cube, so a uniform alphabet spends most of its symbols on directions the hand never visits. DigitCode‑A fits the codebook to data by spherical k‑means and keeps the bone as the unit.

14.71° → 8.45° at the same 26 symbols

02 · group

A finger's bones move as one chain

Mutual information averages 39% within a finger and far less across fingers. DigitCode‑F concatenates a finger's four bones and quantizes them jointly, moving the unit from the bone to the finger and cutting along the weak cross-finger seam.

5.50° at one third of the rate

03 · layer

One finger token caps precision

A single code must summarize four bones. DigitCode‑H keeps both units at once: the finger code as a coarse “which pose,” a per-bone code on its residual as “how much.” The finger is the interface unit, the bone the precision unit.

3.26° at HL's rate, continuing to 1.86°

Results

The unit sets the frontier, not the quantizer

At a fixed unit, swapping the quantizer moves reconstruction by at most 0.10°—inside the learned VQ's own ±0.04° seed noise. Adding quantizer depth at a fixed bone unit costs 0.38°. Changing the unit of the coarse stage—same two-stage mechanism, same fitting code, the per-bone arm given more rate—still leaves the finger unit 1.63° ahead. A grouping control at identical block length, codebook size, and rate confirms the gain comes from the anatomical seam and not from the larger block: regrouping the same bones across fingers gives 8.91°, at random 9.48°, versus 7.10° within finger.

**Table 1. Three-step progression** on held-out InterHand2.6M. Below the rule: a learned VQ, a MoMask-style sequence RVQ‑VAE *system* with temporal context, and a continuous joint-angle baseline. Rate is reported both fixed-width (primary) and as empirical entropy.
Representation	fixed b.	entr. b.	Ang. (°)
HL‑26 (fixed, K=26)	4.70	3.29	14.71
DigitCode‑A (K=26)	4.70	4.38	8.45
DigitCode‑F (K=256)	2.0	1.94	5.50
DigitCode‑H (K₁=128, K₂=8)	4.75	4.37	3.26
DigitCode‑H (K₁=128, K₂=16)	5.75	5.20	2.55
DigitCode‑H (K₁=128, K₂=32)	6.75	6.12	1.86
learned VQ (K=26)	4.70	4.47	8.36
learned VQ (K=128)	7.0	6.70	4.11
MoMask-style RVQ‑VAE (V=4)	1.80	1.67	5.83
MoMask-style RVQ‑VAE (V=14)	6.30	5.99	3.90
joint-angle (continuous)	6.0	5.21	9.06

Rate-distortion scatter with log distortion axis; the three DigitCode families lie below and left of learned, residual and continuous baselines. — **Rate–distortion design space** (held-out InterHand2.6M; log distortion, fixed-width rate). Each family is a sweep over codebook size, so the figure doubles as a sensitivity analysis. Learned, residual, and continuous encodings all appear here—the separation that matters runs along the *unit*, not along the quantizer.

The hierarchy tracks what tasks need

A task's natural unit is the coarsest one that still carries the information it needs: dynamics turn on tiny frame-to-frame velocities and read the bone; interaction turns on which finger is doing what; identity turns on the coarse silhouette of the whole hand. Not every row is a win—forecasting only ties a “predict no motion” baseline, retrieval's spread is modest, and the robot lookup trades accuracy for speed—but the ordering holds, and it holds from the other side too: under estimator noise the coarse HL‑26 symbol is the most robust, because a coarse alphabet quantizes the noise away.

**Table 2. Six downstream uses**, ordered by the anatomical unit their granularity calls for. Full per-tokenizer tables and protocols are in the paper's appendix.
Unit	Reads	Task	Key number
bone	dynamics	Distribution generation	AE‑FGD 3.01 vs. HL‑26 4.46 · speed‑JS 2.2×
bone	dynamics	Single-step forecasting	12.90° (−31% vs. HL‑26; continuous copy 12.74°)
finger	interaction	Retrieval & editing	P@1 25.4% (23× random) · off-finger 0° (MANO edit: 0.90 violations)
finger	interaction	Denoising & repair	+5.5° @ σ=20° (MANO refit ~440× slower) · ASL 14.03°→4.29°
hand	identity	Gesture classification	top‑1 49.7% vs. HL‑26 43.0% (+6.7pp, CI [+4.3, +9.0])

What a genuine unit buys

Because a finger is a complete, enumerable unit, its token is not only more accurate—it is an interface. Three things follow with no added machinery:

Detect and repair a malformed finger. The distance from a finger to its nearest codeword flags corruption with zero training, and is the only detector strong in both failure regimes (Table 3). A masked transformer then rewrites only the flagged finger, conditioned on the other four.
Compile retargeting onto a robot hand. A per-finger codebook is a finite set, so the map from code to robot joints is solved once—640 IK solves (5 fingers × 128 codes), 1.6 s—into a lookup table. Streaming retargeting then costs an O(1) table assembly: ~3 orders of magnitude faster than per-frame optimization, within 0.7 mm on Allegro. The finger's contribution here is addressability and amortization, not peak accuracy.
Recombine into unseen handshapes. Fit on FreiHAND and tested on ASL handshapes—novel finger combinations—a per-finger encoding degrades less (31.2°) than a 6.4× larger whole-hand codebook (34.9°), even though that whole-hand code is more accurate in-distribution.

**Table 3. Corrupted-finger detection** (AUC, higher is better) in two regimes: off-manifold perturbation and anatomically illegal poses. The codeword residual needs no training; a trained Mahalanobis detector edges it off-manifold but collapses on illegal poses, and a MANO-refit residual has no detection power at all—distance to a closed parametric family is not a validity signal.
Detector	off-manifold	illegal
codeword residual (ours, zero-training)	0.953	0.823
Mahalanobis (trained)	0.961	0.684
intra-finger incoherence	0.737	0.615
MANO-refit residual	0.428	0.534

Interactive companion

Everything below is decoded live from real per-finger k‑means codebooks, built by the same procedure as DigitCode‑F and fit on the same held-out InterHand2.6M partition. They are smaller than the paper's headline configuration (K=64 here, K=256 in Table 1) so that one finger's codewords can be shown and swept by hand; the error you see is therefore an upper bound on the paper's. Nothing here is illustrative.

01 — what a token is

A token is a whole finger, and you can read it

Each per-finger codeword decodes to a complete, anatomically plausible finger pose. Here is one finger's codebook, swept from extended to curled — twelve of the poses a single token can name. Because a code quantizes the finger jointly, every token is legal by construction; there is no way to write an impossible finger.

← extendedindex finger · 12 codewordscurled →

Enumerable and readable. A learned VQ latent is a finite set too, but its codes name no physical thing and can decode to broken hands. A per-finger direction code names a pose you can point at.

02 — why the unit matters

Edit one finger, and only that finger moves

Pick a finger, then slide its code. With DigitCode the rest of the hand is untouched, exactly — one token, one finger. Switch to a continuous MANO-style edit and the same change spills across the hand, because its parameters are not decomposed per finger. This locality is what turns one token into a handle to detect, repair, and retarget a single finger.

edit view

Active finger

Its code 0

other fingers moved: 0.00°

DigitCode edit is exactly local — a rewrite of one token touches one finger.

The interface results above rest on this. Because the edit is addressable and legal by construction, the same per-finger code can flag a malformed finger in a generated hand, repair it in place, and compile onto a robot hand — things a continuous pose has no addressable part to do.

03 — now you try

Hand Match

game

A target hand is a continuous pose. Tokenize it: pick a code for every finger to match the ghost. You will do by hand exactly what the tokenizer does — and you will find you can never quite reach 0°. That residual gap is the quantization error the tables measure.

reconstruction

targetyou

MATCHED

gesture 1/6

—° mean bone error · best possible —°

Slide each finger to a code. A green code id means that finger is at its best.

your tokens [· · · · ·]