Detecting Smoking and Phone Use in CCTV Footage
A workplace safety project combining YOLO training, person tracking, and video alerts for signs of smoking and phone use.
Overview
In a wide CCTV shot, a phone may be barely visible and a cigarette can occupy only a few pixels. Distance, motion blur, and changes in lighting make those cues easy to miss. For workplace safety monitoring, the system also needs to decide when the evidence is consistent enough to flag an event for review.
I trained the object detectors and built two video pipelines, one for smoking and one for phone use. Both follow people across frames, look for behavior cues within each person's crop, and write an annotated video with tracking boxes and on-screen alerts.
Demonstrations
I sourced these sample clips online to showcase the project’s final inference results. I kept them out of training and trained the detectors on separate datasets of CCTV imagery for real-world workplace safety scenarios.
Smoking detection showcase
Data and Training
Choose labels that preserve context
I worked with two separate datasets: roughly 14,000 images for phone detection and 7,000 images for smoking detection. Before training, I checked annotation consistency and removed invalid labels. For phone detection, I labeled the phone itself. For smoking, I labeled the hand holding the cigarette, up to the wrist. That gave the detector a larger region to learn from and preserved the grip and hand shape when the cigarette was hard to see.
Test head labels during training
I trained separate Ultralytics YOLO detectors and compared versions with and without a head class. My hypothesis was that learning to detect heads alongside phones or smoking-related hands could help the model learn the visual context around each target. The archived runs also varied other training settings, so those experiments leave the effect of head labels on their own unresolved.
Video Pipeline
- Track peopleDetect people in the full frame and assign track IDs across frames.
- Inspect each cropExpand each person box and search the crop for the phone or cigarette-holding hand.
- Stabilize detectionsTrack behavior cues within each person crop and smooth the displayed boxes.
- Check and alertCheck how long the cue persists for each person and draw alerts on the output video.
The person crop restricts the search to a likely region and lets the second detector devote its input resolution to that person. I expanded the crop slightly to give hands near the edge of the person box room to stay in view.
I kept the alert decision separate from the frame-by-frame detector. A cue has to persist for a set duration before the system shows an alert. A brief missed detection can leave the timer running because blur or occlusion can hide a cue for a few frames. A longer gap resets it. For phone use, persistence filters out isolated detections, though a phone held in view can still trigger an alert even when the person is simply carrying it.
Limitations and Next Steps
The second stage depends on finding the person first. If the person detector misses someone in a crowded or distant view, the behavior detector has no crop to inspect. Even inside a crop, a hand or phone can be hidden, and some gestures resemble the target behaviors from certain angles.
The next step would be to merge the datasets and review them for missing cross-labels, such as a phone in a smoking image. I would then compare phone-only labels with labels that include the hand holding the phone, and test a unified detector. One proposed alert check would use overlap between the head and hand boxes as an additional cue for active use. Its value would need to be tested across camera angles and against clips of people simply holding a phone.