Embodied AI system project

Pekingman

A modular agent stack exploring memory, perception, task planning, and Unity-based interaction for an artificial life simulation.

Watch demo

Project demo

Pekingman v1.0.0: A Multimodal AGI-Oriented Agent System

TingYun Yin, Massachusetts Institute of Technology (MIT) Peixu Cai, University of Southern California (USC) Shahzaib Saqib Warraich, University of Southern California (USC) Osama Fawad, Université Bourgogne Europe (UBE)

Pekingman v1.0.0 is a multimodal AGI-oriented agent system designed for environmental perception, long-term memory, human-like reasoning, emotional consistency, and real-time behavioral response. Equipped with a high-capacity persistent memory system, it can store, retrieve, and revisit historical events, giving NPCs authentic human-like behavioral logic.

System shape

A shared pipeline for agent state and action

Memory

Working and long-term memory files preserve agent context across runs.

Perception

Input folders and collection utilities bring text, video, and environment signals into the system.

Execution

Module orchestration, task selection, and Unity-facing queues connect decisions to visible behavior.

Functionalities

What the system is built to do

Observe

Collects text, video, and environment input so the agent can interpret current conditions.

Remember

Maintains working state and long-term context to support continuity across decisions.

Plan

Turns goals and sensed conditions into selected tasks, module activity, and next actions.

Act

Connects planned behavior to simulation-facing controls for visible embodied interaction.

Monitor

Provides a web panel for inspecting module progress, shared state, and system activity.