Assignment 3: Drone Detection and Tracking
Overview
For this assignment, I built a drone detection and tracking pipeline that works on video files placed in the data/videos/ folder. The pipeline first extracts frames from each video, then runs a trained YOLO detector on those frames, and finally tracks the detected drone over time using a Kalman filter.
The goal of the project was not only to detect the drone in each frame, but also to maintain a stable track when the detector briefly misses it. The final output includes detection frames, saved detection data, and tracked videos with a trajectory drawn over time.
Final Pipeline
The final pipeline is:
- Extract frames from all videos in the input folder.
- Run drone detection on the extracted frames.
- Save the highest-confidence detection for each frame.
- Use the center of the detection bounding box as the Kalman filter measurement.
- Predict the drone position through short missed detections.
- Write output videos with:
- the detection bounding box
- the tracked center point
- the drone trajectory line
Dataset Choice
Final dataset used for submission
At one point, I also tested a second dataset after noticing false positive detections on clouds in one of the harder test videos. I hoped a different dataset would improve generalization. However, after comparing outputs from both approaches, the original dataset combined with higher-resolution detection performed better overall on the provided videos. Because of that, I kept the original dataset and model for the final submission.
Second dataset tested during experimentation
Why I kept the original dataset
The original dataset gave better practical results on my actual test videos, especially when combined with a larger detection image size. The newer dataset was a useful experiment, but it did not improve the final tracking output enough to justify switching the full pipeline.
Test Videos
For the assignment, I used the two required YouTube videos as the main test inputs:
These were the two videos I downloaded, processed into frames, ran detections on, and then used for the final Kalman filter tracking outputs.
Detector Configuration
I used a YOLO-based detector for the drone detection stage.
Final detector setup
- Model family: YOLO
- Final trained model:
drone_yolov8s_1280_b8
- Training image size: 1280
- Epochs: 30
- Batch size: 8
- Frame extraction rate: 20 FPS
- Detection confidence threshold: 0.45
- Detection image size at runtime: 1920
- Detection output: highest-confidence detection per frame
Why these settings were chosen
I initially started with more basic settings, but one of the test videos contains a very small and distant drone. Because of that, larger image sizes worked better than lower ones. Increasing the detection image size helped preserve more detail for small-object detection.
I also raised the confidence threshold to reduce obvious false positives.
Highest-confidence detection choice
For the final submission, I simplified the per-frame detection logic by using only the highest-confidence detection in each frame. This made the pipeline more straightforward and easier to integrate into the Kalman tracking stage.
Frame Extraction
Frames are extracted from every video found in the data/videos/ folder. This makes the code work for more than just the two sample videos.
Extraction settings
- Input: all supported video files in
data/videos/
- Output: extracted PNG frames in
data/frames/<video_name>/
- FPS used: 20
I used PNG frames instead of JPG to avoid extra image compression during extraction. This was helpful because the drone is very small in some frames.
Kalman Filter Design
The tracking stage uses a Kalman filter to estimate the drone’s motion over time.
State vector
I used a 4-dimensional state:
cx = center x-coordinate of the bounding boxcy = center y-coordinate of the bounding boxvx = velocity in xvy = velocity in y
So the full state is:
[cx, cy, vx, vy]
Measurement
The measurement is the 2D center of the detected bounding box:
[cx, cy]
The detector provides the current observed center, and the Kalman filter smooths it over time and predicts future positions.
Motion model
I used a constant-velocity motion model. This assumes the drone’s movement between nearby frames is approximately smooth, which is reasonable at 20 FPS.
Kalman Filter Noise Parameters
The Kalman filter was configured with the following core matrices.
State covariance P
I initialized the covariance with a fairly large value:
This reflects high uncertainty when the tracker starts.
Measurement noise R
I used:
This means the detector measurements are treated as somewhat noisy but still reliable enough to update the track.
Process noise Q
I used a moderate process noise matrix:
- lower noise for position
- slightly larger noise for velocity
This helps the filter remain stable while still allowing motion changes.
Why these values were chosen
I wanted the tracker to be smooth, but not so rigid that it could not follow motion changes. These values worked reasonably well for the sample videos.
Tracking Logic
The tracker starts when a detection is available. After initialization:
- if a detection exists in the current frame, the Kalman filter updates using the detection center
- if a detection is missing, the Kalman filter predicts the next position without updating
This lets the track continue for a short time even when the detector misses the drone.
Missed detection handling
I used a max_missed setting so the tracker can continue through short-term missed detections.
For the final version:
At 20 FPS, this allows the tracker to continue for about 0.25 seconds without a detection before stopping.
This was important because the detector sometimes misses the drone when it is very small or blended into the background.
Output Files
The pipeline generates several outputs.
Detection outputs
- Annotated detection frames in
data/detections/<video_name>/
- Detection data in Parquet format in
outputs/parquet/
Tracking outputs
- Final tracked videos in
outputs/tracked_videos/
The tracked videos show:
- the detection bounding box
- the tracked center point
- the trajectory of the drone over time
Failure Cases
The hardest part of this assignment was handling small drones against a cloudy sky.
Main failure case
The most noticeable failure case was in the harder video where the drone is far away and very small. In some frames, the detector incorrectly labeled parts of a cloud as a drone. This happened because the cloud texture sometimes looked more confident to the detector than the actual drone.
Other difficult cases
- drone is extremely small in the frame
- low contrast between the drone and background
- cloud textures that look like objects
- brief detector misses when the drone moves or becomes hard to see
How the Tracker Handles Missed Detections
The tracker helps when the detector fails for a few frames in a row.
Behavior
- when detections are present, the track follows the measured center
- when detections disappear briefly, the Kalman filter predicts the next location
- after too many missed frames, the tracker stops writing frames until detections return
This keeps the output smoother and avoids the track disappearing immediately after one missed detection.
Limitation
If the detector locks onto the wrong object, such as a cloud, the Kalman filter will still track that wrong object because it only smooths the detector output. In other words, the tracker improves temporal consistency, but it cannot fully correct major detection errors by itself.
Final Notes
For the final submission, I chose the version of the model and dataset combination that gave the best actual visual tracking results on the provided test videos, even though I also experimented with a second dataset. The final pipeline is simple, modular, and works on all videos in the input folder instead of only the provided examples.
