Overview

Traditional puppet manipulation often places the operator close to the performance area, creating a crowded setup and limiting where the puppet can be positioned. This paper presents a compact, cost-effective robot that moves the operator away from the puppet while retaining real-time control.

The system combines an adjustable mechanical platform with a custom YOLOv8 pose model. It observes a manually operated puppet, estimates seven keypoints, filters the detected motion, and sends joint commands over Wi-Fi so the robotic puppet can imitate the reference motion. Education, entertainment, and teleoperation are proposed applications; the paper does not report a clinical or classroom study.

Demonstrations

Architecture

The prototype forms an end-to-end perception and control loop rather than using pose detection as a standalone demonstration.

  1. Reference motionA human operator moves a puppet within the camera view.
  2. Pose estimationYOLOv8 detects a bounding box and seven puppet keypoints in each frame.
  3. Motion filteringKalman and threshold-based filters reduce jitter and reject abrupt changes.
  4. Joint mappingRelative keypoint positions are converted into target angles for the head and arm joints.
  5. Wireless actuationCommands travel over Wi-Fi to the NodeMCU, which updates the servo positions.

The current vision pipeline drives the robot's head and arms. Although the hardware also includes full-body rotation and can attach to a mobile base, those motions are not yet controlled by the trained model.

Mechanical design

We made the stationary robot adjustable so it can fit puppets of different sizes. With its arms retracted, it measures about 160 mm high, 90 mm wide, and 75 mm deep; its arms are 80 mm long and its head structure is 50 mm.

Most structural parts are 3D-printed in PLA, while Plexiglas provides a rigid base. Two SG90 servos move the arms, a third SG90 moves the head, and an SG-5010 Pro servo rotates the body. The modular construction keeps the motors accessible for maintenance and allows the stationary unit to be mounted on an optional base with two DC motors and three wheels.

Model and results

The custom dataset contains 2,000 puppet images annotated in Roboflow with a bounding box and seven keypoints: head, neck, left and right shoulders, left and right wrists, and pelvis. These points mirror the robot's articulated structure and provide the geometry needed to map observed motion to its joints.

Rotation, resizing, and brightness augmentation were used to diversify the training data. The learning rate and batch size were tuned during training, while bounding-box and pose losses were tracked on both training and validation data.

We reached 91.4% overall accuracy and 0.91 mean average precision, alongside a false-positive rate of 0.08, a false-negative rate of 0.12, and a mean squared error of 0.015. These are the metrics reported for this prototype, not a cross-model benchmark.

MetricValue
Overall Accuracy91.4%
False Positive Rate (FPR)0.08
False Negative Rate (FNR)0.12
Mean Squared Error (MSE)0.015
Mean Average Precision (mAP)0.91
Training Loss0.082

Control

Detected keypoints are converted into commands for the physical robot. The distance between the neck and wrist points informs the shoulder positions, while the relative head and neck positions control the head servo. A Kalman filter smooths frame-to-frame variation, and threshold-based filtering rejects changes that are too abrupt to represent plausible motion.

After filtering and joint mapping, the processed commands are transmitted over Wi-Fi to the NodeMCU. Servo targets are updated as new frames arrive, allowing the prototype to respond continuously within the limits of its current mechanical and vision configuration.

Limitations

  • One-puppet dataset: The model was trained on a single puppet, limiting evidence of generalization to different puppet shapes and appearances.
  • Partial motion mapping: The trained system controls the head and arms but does not infer whole-body rotation or mobile-base motion.
  • Planned extensions: We want to add rotation information, expand the dataset, improve augmentation, and explore pruning or quantization for edge deployment.