What if reality is a simulation? Every generation asks it again with whatever
machinery it has. George Hotz asked
it at SXSW
the way an engineer would:
“Can we get out? Meet God? Kill him?”(via Roman Yampolskiy, “How to Hack the Simulation?”)
The question has pulled at me for years, and the only honest response I have is to try to
build one. Nobody funds that outright. So I build intermediaries that are useful on their
own terms, and each one points more energy at the question than I have alone.
The work starts in autonomous driving. Simulated worlds there look photoreal, but the
people in them are boxes on rails: no gesture, no hesitation, no intent. A car has to read
people to be safe, which makes believable humans a real and funded problem, and that gives
me room to study humans properly. I start below language, with movement: how a body walks,
hesitates, reacts. Each chapter below leaves something useful behind and adds a layer to the
same simulated person: reaction, perception, identity, and one day a mind…
Generating human motion from natural language: a model over a continuous
motion space, trained on real captured behaviour.
Dataset · recorded first-hand, with real participants488 street-crossing sequences15 participants8 long-tail behaviours
01 · Walk-the-Talk
Scripted pedestrians cover only the nominal cases: nobody stumbles drunk, nobody
recoils from a near miss. We recorded those behaviours in motion capture with real
participants and trained a language-to-motion model on them. Retargeted into
CARLA, a written phrase becomes the long-tail traffic data autonomous driving needs.
Figure 1 · generated long-tail behaviours, retargeted into CARLA.
IEEE IV 2024
02 · beyond the recorded vocabulary
The paper model draws from a fixed vocabulary of recorded
behaviours. The live engine goes further: diffusion denoises a continuous motion
space, steered by your words, so it responds even to prompts it was never trained on.
Try it:
Livespeak, and it movesengine: connecting…
Your voice is transcribed by Whisper on the same GPU;
only text enters the engine.
State-of-the-art driving agents are trained and ranked on public
leaderboards where every pedestrian follows a script. HABIT replays 4,730 real human motions
into the simulator, and the same agents begin to hit people, at rates up to 7.43 collisions
per kilometre. Below: how one real crossing is carried into the simulated world, then the
numbers.
Figure 2one crossing, two worlds
source · Ono Kosuki / Pexels
retarget · HABIT / CARLA
The retargeting is deterministic, so the two bodies move in step.
Only the motion is transferred, not the appearance.
7.43collisions / km on real motion by an agent that scores ∼0 on the leaderboard
The agents are unchanged. The only difference is that the pedestrians
now move like real people. Full tables and metrics are in the paper.
framing
§3 · Echoes
Step into frame.
No suit, no markers: a camera lifts you into the model, live.
idle
§4The frontier
What is happening next.
This section changes as the work does. These are the threads open right now.
01 · reaction
A person responds to the world.
Pedestrians signal intent through the whole body long before they step off
a curb: a head turn, a raised hand, a lean. Simulators discard all of it. This system
generates the response itself: full-body motion conditioned on the car’s
trajectory, recovered through SLAM and multimodal sensing in real driving logs, and
learned from real interactions in nuScenes, the Waymo Open Dataset, and our own
autonomous car, AVA.
an example of vehicle-conditioned pedestrian motion · hood-cam left, the same nine seconds from above rightthe approach · the car’s trajectory, encoded, conditions each denoising step of the same latent motion engine from §1 · full architecture in the paper
An early prototype already walks. A camera bolted to the head is the
eye. A detector reads the street from that eye, a depth pass tells it how far things
are, and the result becomes its own decision to cross: control on the walker,
no script, no replay.
the eye · head-mounted camera
what it sees · crosswalk, 0.81
how far · metric depth
its own senses · camera, detection and depth from head height · prototype in CARLA
one autonomous crossing · four moments, its own view inset
zₜ₊₁ = f(zₜ, iₜ) · xₜ = g(zₜ, iₜ, eₜ)
z intent (latent) · i interaction: distance, closing speed, time-to-contact · e viewpoint · x the observed motion
The reaction depends on interaction variables, not on the car as an
object. Moving the camera changes only x; the decision z is unaffected. Modelling the
decision separately is what lets the same person transfer across scenes and
viewpoints.
The missing half is a mind that carries memory.
Interactive Simulacra of Human Behavior (Park et al., 2023) put 25 language
agents with memory, reflection and planning in a toy town, and they organised a party no
one scripted. Those agents had no bodies; the walkers above have bodies and no minds yet.
The plan is to combine the two inside CARLA: the memory, reflection and planning stack
driving the embodied pedestrians above.
research in progress · working prototype in CARLA
03 · identity
A body described in words.
one photo, measured
Once it moves, sees and acts, it needs to be someone. One camera
recovers a body; then it is reshaped by its own local dials: shoulder width,
waist, hip, thigh girth, leg length, each a real tape-measure dimension, not an
anonymous shape number, with height, mass and BMI re-measured live as it changes. This is
also why the shape space moved from SMPL to ANNY:
SMPL’s shape components are statistical directions with no names, while ANNY’s
parameters are the measurements themselves. That one measurable body is what everything
downstream reads: what fits you (try-on), how you’re changing (body
composition), how you move (rehab & range of motion), and who you are
online (a rigged avatar).
venture direction, in progress
04 · synthesis
One person, eventually.
Shape and pose are the surface. A real body obeys forces: bones scaled to
the person, mass in the right places, ground pushing back at every step. Tools like
AddBiomechanics and
Nimble Physics already turn raw
motion capture into physically consistent, person-scaled skeletons; that layer belongs
under everything above. Beyond it, the modelling gets more biological: muscle activations
instead of joint torques, reflexes instead of replayed motion, memory instead of state
variables.
a real body running · the centre of mass, and the ground pushing back at each footfall · in flight, nothing pushes
Everything above runs while you read this. Each section is a
working layer of the same simulated person: a body that moves, motion that holds up
against reality, a mirror anyone can step into, reaction and perception taking shape.
What is still missing is the mind, and the question at the top of this page is the
reason to keep building.
to be continued · next: muscle models over torque, memory over
state, first experiments toward a mind · this page updates as they land
Papers
HABIT: Human Action Benchmark for Interactive Traffic in CARLA
@misc{ramesh2025habit,
title = {{HABIT}: Human Action Benchmark for Interactive Traffic in {CARLA}},
author = {Ramesh, Mohan and Azer, Mark and Flohr, Fabian B.},
year = {2025},
eprint = {2511.19109},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
note = {Accepted to WACV 2026}
}
type: Newsreader & IBM Plex Mono · rig: Y Bot (Adobe Mixamo) ·
shape model: ANNY (NAVER, Apache-2.0) · video codec: ASCILINE ·
the live pieces run on the author’s own GPU ·
hand-built, no framework · rev d3 · 2026-07-03
