2

Anomaly Detection in Surveillance Video with Convolutional Autoencoders

Unsupervised convolutional autoencoder system for detecting and localizing forbidden vehicles in pedestrian-only spaces using reconstruction error heatmaps.

Website

Overview

This project implements a convolutional autoencoder–based anomaly detection system to identify forbidden vehicles entering pedestrian-only spaces in surveillance video.

Instead of relying on labeled anomaly data, the model is trained in an unsupervised setting to reconstruct normal pedestrian scenes.
Frames containing anomalous objects (bicycles, carts, skateboards, vehicles) produce high reconstruction error, which is used to detect and localize unsafe events.

The system generates both:

  • Frame-level anomaly scores for real-time alerting
  • Pixel-level heatmaps highlighting where the anomaly occurs

This design supports fast, interpretable monitoring in safety-critical environments such as university campuses and public walkways.

Problem Context

Pedestrian-only areas occasionally experience intrusion by wheeled vehicles, creating collision risk and safety hazards.

Operational constraints:

  • False negatives are more costly than false alarms
  • Alerts must be fast and actionable at the frame level
  • Operators benefit from visual localization of anomalies

Approach

I built a convolutional autoencoder pipeline in TensorFlow/Keras that:

  • Trains only on normal pedestrian traffic
  • Reconstructs incoming frames and computes per-pixel reconstruction error
  • Aggregates error into:
    • Frame-level anomaly scores
    • Pixel-level heatmaps for localization

Key components:

  • Convolutional encoder–decoder architecture
  • Reconstruction-error–based anomaly scoring
  • Threshold-based event detection
  • Visualization pipeline for triage and debugging

Results

Qualitative Anomaly Localization

Original surveillance frames, reconstructed outputs, and reconstruction error heatmaps highlighting anomalous vehicles detected by an autoencoder.

Reconstruction and anomaly localization using a convolutional autoencoder.
Top row: original surveillance frames.
Middle row: reconstructed frames learned from normal pedestrian traffic.
Bottom row: reconstruction error heatmaps highlighting anomalous vehicles entering pedestrian-only areas.

The autoencoder successfully reconstructs normal pedestrian scenes while producing high reconstruction error in regions containing anomalous vehicles.
This enables both frame-level anomaly detection and pixel-level localization for operator triage and interpretability.

Quantitative Detection Performance

ROC curve for frame-level anomaly detection using an autoencoder with an AUC of 0.742.

ROC curve for frame-level anomaly detection using reconstruction error as the anomaly score (AUC = 0.742).

The ROC curve demonstrates reliable separation between normal and anomalous frames, achieving an AUC of 0.742 without any labeled anomaly training data.

This demonstrates how unsupervised deep learning can support safety monitoring when anomaly labels are rare or unavailable.

Technical Highlights

  • Unsupervised anomaly detection with convolutional autoencoders
  • Pixel-level reconstruction error for localization
  • Frame-level alerting suitable for streaming pipelines
  • Implemented end-to-end in TensorFlow / Keras

Repository

Code and experiments available on GitHub:
github.com/DavidCastroPena/keras-anomaly-autoencoder