Driveway Animal Detector

A personal computer vision project using footage from my own security camera

My driveway camera, a wired Arlo floodlight, sends an alert when it detects an animal. It doesn’t identify which animal, and while reviewing a month of recordings I found it had flagged a moving tree-branch shadow as one. The goal was a pipeline that identifies the animal (rabbit, squirrel, cat, or raccoon), works day and night, and ignores false triggers such as shadows and wet concrete.

Roboflow’s onboarding agent started with SAM 3, a foundation model that detects objects from their names without training, instead of beginning with data labeling. I went with that approach, with the plan to measure the baseline first and add custom work only where it fell short.

This one is separate from my work at Spectra Logic. I scoped the project, supplied the footage and ground truth, reviewed results, and made the decisions. Roboflow’s agent built the workflow and wrote the scripts.

What the data showed

The camera is cloud-locked with no live stream access, so all processing ran on downloaded clips. The footage covers three lighting conditions: daylight, color under the floodlight, and grayscale infrared. The floodlight often switches on partway through a clip.

Across the month of footage, lighting and species never overlapped. Rabbits and squirrels appeared only in daylight. The cats (one black, one white) and the raccoon appeared only at night. A model trained on this data could learn to associate darkness with cats and raccoons instead of learning what the animals look like. SAM 3 wasn’t trained on this footage, so that risk didn’t apply here, but it would need to be addressed before any fine-tuning.

Baseline results

I set aside four test clips: a daytime squirrel, a nighttime raccoon, a nighttime cat moving along the landscaping, and the shadow clip the camera had flagged.

All three animals were identified correctly. The squirrel was detected at about 90 percent confidence near the bottom edge of the wide-angle frame. Under infrared, the raccoon’s label varied between raccoon, cat, and squirrel from frame to frame, but most frames labeled it raccoon.

On the shadow clip, the raw model output contained no animal detections across all 269 processed frames.

From detections to visits

A useful log records visits, not individual frames. The agent built a workflow that runs the detector, tracks each animal across frames, and records one visit per track using the most frequent label, with a minimum of three detections required for a visit to count.

The squirrel clip was tested first, since a clip with a known animal should produce a visit. It didn’t. Detections were coming through, but the visit counter stayed at zero. Had the shadow clip been tested first, the broken counter would have produced what looked like a correct result.

After that fix, the cat clip showed two further issues. The tracker split the cat into three separate tracks, likely because plants blocked it from view as it moved along the landscaping. I set a rule to treat detections of the same species separated by roughly three seconds or less as one visit. The gap in that clip measured 3.07 seconds, and the cutoff was set at 3.1. That value was based on a single clip and still needs testing against more footage, particularly clips with two animals close together.

Where it stands

The cat clip still produced no visit. The agent traced the cause to how Roboflow’s hosted preview processes video: it handles batches of frames concurrently, so multiple frames read the same running tally and overwrote each other’s results. Individual frame detections were reliable, but counts accumulated across frames were not. This also meant the earlier squirrel and raccoon visit results, although correct, could not be relied on.

Processing frames in order resolves this. The agent wrote a script to stream clips sequentially from my laptop, which ran into two further issues: the script used an SDK function not present in the installed version, and my free plan does not include the serverless access the script required. The preview also does not export a full clip’s frame-by-frame results.

The agent then removed the visit counts from the workflow so it would not report numbers that might be wrong, which I agreed with. The visit logic now exists as a separate script that builds visits from ordered frame results and rejects incomplete input, ready for use once sequential processing is available.

What’s next

Run the workflow in a sequential environment, likely locally, and rerun all four clips on a single final version so all published results come from the same build.

Record the vote share for each visit and flag low-agreement visits for human review. The raccoon’s label varied from frame to frame at night, and routing uncertain results to a person is more practical than pursuing perfect accuracy.

Add test clips with two animals close together, more raccoon footage, and any nighttime rabbit footage, to properly test the merge rule and the lighting risk.