AISoccerCoach: Team & Role Assignment
Learning appearance features to group soccer players by team and role.
Developing soccer team and role assignment through triplet learning, 19 archived model versions, and a 15-model development benchmark.
Overview
Assigning teams and roles in soccer video is challenging when uniforms vary between matches and detections contain occlusion, motion blur, or background clutter. This work presents a hierarchical pipeline integrating appearance learning, semantic role interpretation, and temporal label aggregation. A shared MobileNetV2 or ResNet50 encoder is trained using same-video triplets, balanced anchor sampling within each video, and a configurable sampled-negative objective to produce normalized appearance embeddings. During inference, an initial clustering stage organizes participant appearances, while detector-derived cues identify goalkeeper and referee groups. The remaining observations are then reclustered into the two outfield teams. A final consistency stage applies per-track majority smoothing and optional goalkeeper constraints while preserving incoming tracking identities. On a development benchmark of 5,581 crops from 36 video groups, the MobileNet V12 configuration achieved 97.31% mean per-group clustering accuracy after ground-truth label alignment. The architecture provides an offline, modular workflow for converting tracked participant detections into local team and role annotations.
Demonstrations
The showcase plays saved predictions across four clips. The inference walkthrough follows one tracked frame from crops to final labels. The training walkthrough shows how the encoder learned appearance similarity. All three use MobileNet V12 artifacts. The development benchmark below evaluates crop clustering separately.
Results showcase
Problem
A traditional approach is to estimate jersey colors from player crops using tools such as OpenCV, then group players by color. Changes in lighting, shadows, and camera exposure can make the same kit look different across frames. In small or blurred crops, the pitch or a player in front can account for more pixels than the jersey itself.
The groups also depend on the match. A team can wear a different kit in the next video, while goalkeepers and referees form smaller groups alongside the two outfield teams. The task is to learn which crops share a team or role within a match, so two different teammates can form a positive training pair.
Method
- Tracked cropsRead external boxes and IDs, then resize valid person crops.
- Appearance embeddingsRepresent each crop with a learned, normalized feature vector.
- Team and role groupingInterpret role clusters with detector hints and split remaining people into two teams.
- Label consistencyApply per-track majority smoothing and an optional goalkeeper constraint.
Embed externally tracked observations
The pipeline reads a video interval with frame-aligned boxes and IDs from an external tracker. It resizes valid crops to 124 × 124 pixels, converts them to RGB, and scales pixel values to [0, 1]. The encoder produces normalized appearance vectors in batches.
Interpret roles and form two teams
The first K-means pass forms up to four appearance clusters. If the most frequent detector label in a cluster is goalkeeper or referee, the cluster receives that role. A second pass splits the remaining crops into two teams. Mean horizontal image position determines the clip-local Team A and Team B labels. Goalkeepers receive a role only.
Smooth labels within incoming tracks
Offline processing uses the full selected interval. When a label reaches the configured majority threshold, the pipeline applies it across that track and preserves the original ID. An optional rule uses detector hints to retain at most one goalkeeper per frame.
Implementation
The package separates training, crop evaluation, and video assignment. An optional adapter supplies tracking inputs. The assignment run saves labels, tracker IDs, embeddings, diagnostics, a silent overlay, and the configuration with input and model hashes. Later consolidation verified model loading, reproduction of saved labels, and complete video decoding.
Training
Data and same-video sampling
I built a dataset with 64,780 training crops from 107 video groups and 1,720 validation crops from 14 groups. Each crop was labeled as one of the two outfield teams, goalkeeper, or referee. I split the data by match, so footage from any given match goes into only one split.
I used triplets to teach the model which appearances belong together. The anchor is the reference crop, the positive is another crop from the same team or role, and negatives come from other classes. Sampling within one video keeps those labels meaningful as kits change between matches. I gave each class a turn as the anchor so the more numerous outfield crops would not crowd out goalkeeper and referee examples.
Architecture and objective
I compared MobileNetV2 and ResNet50 to explore the tradeoff between clustering accuracy and compute. Both encoders start from ImageNet weights and are fine-tuned on 124 × 124 RGB crops. Global average pooling and a learned projection turn each crop into a compact vector, and L2 normalization puts the vectors on a common scale. MobileNet V12 produces 512 features per crop and ResNet V6 produces 256.
Triplet loss trains the encoder to place the positive closer to the anchor than the nearest sampled negative by a margin. Sampling three negatives in later runs gave the loss more chances to encounter a confusing example. I monitored training and validation loss to see how the objective behaved on each split.
How the experiments evolved
- Role examples and separation. Early MobileNet runs confused goalkeepers and referees. I expanded the examples for both roles and tested a larger margin. The mean benchmark score was 58.72% for V4 and 92.34% for V5.
- Embedding size and training length. I varied the margin, training length, and vector size to see how much capacity the task needed. Scores fluctuated across V6–V8 before V9 reached 95.05% with 512 dimensions.
- Negative sampling and training budget. V10 and V11 tested three negatives and scored below V9. V12 kept three negatives and used larger batches and more steps per epoch. That combination gave the highest mean across groups at 97.31%.
- Backbone comparison. I ran the ResNet50 experiments alongside MobileNetV2. ResNet V6 reached 95.91% by group mean and came within one correct crop of V12 when pooling all crops.
Complete training history — 19 model versions
The table brings together the encoder dimensions and training settings for both model families.
| Model | Dim. | Margin | Negatives | Batch | LR | Epochs | Steps / epoch |
|---|---|---|---|---|---|---|---|
| MobileNet V0 | 128 | 0.2 | 1 | 8 | 0.0001 | 5 | 100 |
| MobileNet V1 | 128 | 0.2 | 1 | 8 | 0.0001 | 10 | 100 |
| MobileNet V2 | 128 | 0.2 | 1 | 8 | 0.0001 | 20 | 100 |
| MobileNet V3 | 128 | 0.5 | 1 | 8 | 0.0001 | 50 | 100 |
| MobileNet V4 | 128 | 0.2 | 1 | 8 | 0.0001 | 10 | 100 |
| MobileNet V5 | 128 | 1.0 | 1 | 8 | 0.0001 | 20 | 100 |
| MobileNet V6 | 128 | 1.0 | 1 | 8 | 0.0001 | 20 | 100 |
| MobileNet V7 | 128 | 1.2 | 1 | 8 | 0.0001 | 50 | 100 |
| MobileNet V8 | 256 | 1.0 | 1 | 8 | 0.0001 | 50 | 100 |
| MobileNet V9 | 512 | 2.0 | 1 | 8 | 0.0001 | 50 | 100 |
| MobileNet V10 | 256 | 1.0 | 3 | 8 | 0.0001 | 30 | 100 |
| MobileNet V11 | 512 | 2.0 | 3 | 8 | 0.0001 | 30 | 100 |
| MobileNet V12 | 512 | 2.0 | 3 | 16 | 0.0001 | 20 | 1000 |
| ResNet V1 | 256 | 1.0 | 1 | 16 | 0.0001 | 40 | 100 |
| ResNet V2 | 128 | — | — | — | — | — | — |
| ResNet V3 | 128 | 1.0 | 1 | 16 | 1e-05 | 40 | 100 |
| ResNet V4 | 200 | 1.5 | 1 | 16 | 1e-05 | 40 | 100 |
| ResNet V5 | 256 | — | — | — | — | — | — |
| ResNet V6 | 256 | 1.5 | 1 | 16 | 1e-05 | 15 | 1000 |
Checkpoints and qualitative inspection
During training, validation triplet loss selected the training checkpoint, while a kNN diagnostic on the validation embeddings selected the standalone encoder. Those signals can favor different epochs. The consolidated training workflow selects both exports using validation triplet loss so they come from the same epoch.
I used t-SNE to inspect how the learned features grouped similar appearances. The V12 example shows two large team groups and two smaller role groups. In the labeled panel, Class 3 denotes referees.
Results
15 trained models were compared on a development benchmark of 5,581 crops from 36 video groups. Each model was scored on the same crops so I could compare how well its embeddings separated teams and roles.
What the metric measures
K-means is used to cluster each video group's embeddings into four groups. Cluster IDs are arbitrary, so I used Hungarian matching to align them with the labeled teams and roles before counting correct crops.
Mean per group gives each of the 36 groups equal weight. Pooled crop accuracy divides the total number of correctly aligned crops by 5,581, so larger groups contribute more. Both metrics use the saved cluster assignments and measure crop grouping.
I used the benchmark to compare settings and select a model. Its images were held out from training, and 136 crops also appear in validation. Independent match-level evaluation is pending.
Three candidates, different tradeoffs
| Model | Mean per group | Pooled crops | Worst group | Correct crops |
|---|---|---|---|---|
| MobileNet V12 | 97.31% | 97.06% | 76.61% | 5,417 / 5,581 |
| ResNet V6 | 95.91% | 97.04% | 71.43% | 5,416 / 5,581 |
| MobileNet V9 | 95.05% | 94.84% | 82.71% | 5,293 / 5,581 |
I use MobileNet V12 in the demonstrations because it scored highest when each video group counted equally. That metric gives performance on small groups the same weight as performance on larger ones.
ResNet V6 is almost tied when crops are pooled: V12 has 5,417 correct crops and V6 has 5,416. Their 1.40 percentage-point difference in group means therefore corresponds to just one crop overall. V12 wins 13 groups, V6 wins seven, and 16 tie; different group sizes and complementary failures explain why the summaries differ.
For example, V12 scores 76.61% on v141_difficult while V6 scores 100%. On v102, V12 scores 98.33% while V6 scores 71.67%. The two encoders handle different groups well.
MobileNet V9 has the strongest worst-group score, 82.71%, despite its lower mean. It ranks first on this benchmark if performance on the weakest group is the priority.
Complete benchmark comparison — 15 models
Ordered by equal-group mean. Every row uses the same 36 groups and 5,581 crops, with ground-truth cluster alignment performed independently within each group.
| Model | Mean per group | Pooled crops | Worst group | Correct crops |
|---|---|---|---|---|
| MobileNet V12 | 97.31% | 97.06% | 76.61% | 5,417 / 5,581 |
| ResNet V6 | 95.91% | 97.04% | 71.43% | 5,416 / 5,581 |
| ResNet V4 | 95.39% | 94.97% | 77.97% | 5,300 / 5,581 |
| ResNet V3 | 95.34% | 95.81% | 70.37% | 5,347 / 5,581 |
| MobileNet V9 | 95.05% | 94.84% | 82.71% | 5,293 / 5,581 |
| ResNet V5 | 93.19% | 92.99% | 69.64% | 5,190 / 5,581 |
| ResNet V2 | 92.83% | 92.78% | 54.63% | 5,178 / 5,581 |
| MobileNet V11 | 92.43% | 92.22% | 74.07% | 5,147 / 5,581 |
| MobileNet V5 | 92.34% | 91.79% | 72.22% | 5,123 / 5,581 |
| MobileNet V6 | 92.23% | 92.58% | 70.37% | 5,167 / 5,581 |
| MobileNet V7 | 92.19% | 91.83% | 70.37% | 5,125 / 5,581 |
| MobileNet V10 | 91.91% | 91.36% | 66.07% | 5,099 / 5,581 |
| MobileNet V8 | 90.88% | 90.77% | 68.75% | 5,066 / 5,581 |
| ResNet V1 | 85.92% | 85.88% | 62.11% | 4,793 / 5,581 |
| MobileNet V4 | 58.72% | 57.41% | 37.30% | 3,204 / 5,581 |
Limitations
- Appearance ambiguity: Similar uniforms, small crops, blur, and background pixels can merge classes or split one class into several clusters.
- Role assumptions: Role interpretation depends on detector hints and the initial cluster structure. A clip may contain fewer appearance groups than expected, and the single-goalkeeper option can suppress a valid second goalkeeper.
- Local semantics: Team A/B numbering is clip-local. Goalkeepers have no team assignment, and the output provides no calibrated assignment confidence.
- Upstream tracking: Missing boxes and ID switches affect both grouping and smoothing. Incoming IDs are preserved rather than corrected.
- Offline processing: Whole-interval clustering and smoothing can use future observations. No controlled real-time throughput claim is made.
Future work
Test alternatives for clips with missing roles, similar kits, and two visible goalkeepers. Evaluate the smoothing and goalkeeper rules with controlled comparisons, and investigate assigning goalkeepers to teams. Any confidence or rejection policy should be validated against held-out data before being presented as reliable.
Interested in the code, data, or implementation details? Email me to discuss access.