Analysis of spatio-temporal representations for robust footstep recognition with deep residual neural networks

This 2018 IEEE TPAMI paper verifies identity from footsteps recorded by a floor sensor, using deep residual neural networks. It evaluates a large footstep database from more than a hundred people.

Published in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) in 2018, the paper by Omar Costilla-Reyes, Rubén Vera-Rodríguez, Patricia Scully, and Krikor B. Ozanyan proposes spatio-temporal footstep representations learned from floor-only sensor data for automatic biometric verification. A biometric system checks that a person is who they claim to be from a physical or behavioral trait. Footsteps are a behavioral trait that a floor sensor can capture without a camera and in darkness. The models are trained to separate the fine-grained variability of footsteps between legitimate users of a system, called clients, and impostors.

The footstep data come from a floor sensor made of two rectangular arrays of piezoelectric sensors, which record how pressure is distributed under every step. Each footstep is represented in two ways, as a single spatial pressure image and as a sequence of frames over time, and each representation is passed to a deep residual network, a convolutional network with skip connections that make deep models easier to train. The class scores of the two streams are then fused, as the architecture figure shows. The study evaluates a large footstep database from more than a hundred people, organized with a large cohort of impostors and a small set of clients, across three scenarios defined by how much training data is available: airport security checkpoints (smallest), workplaces (medium), and homes (largest).

The best configurations reached a very low equal error rate, the setting at which false acceptances and false rejections occur equally often, which the authors report as a large improvement over the previous state of the art. A feature analysis of the residual networks showed clustering of legitimate users’ footstep patterns, evidence that the learned representations capture what distinguishes one person’s way of walking from another’s. Because a floor sensor works unobtrusively, without a camera and in any lighting, the approach suits smart buildings, workplaces, homes, and security checkpoints, the three scenarios the study evaluates. The work sits within Dr. Costilla Reyes’s research on gait analysis for healthcare and security, in which the same floor-sensing and deep-learning methods support both identification and the study of how people walk.

The paper formed the final manuscript of Dr. Costilla Reyes’s PhD thesis and was selected for the University of Manchester’s InAbstract 2018 showcase of new research from the Faculty of Science and Engineering. He presented the work at the ARM Research Summit in Cambridge in September 2017. He was listed as the speaker for a University of Cambridge Computer Laboratory seminar in January 2018 and was invited to give a seminar at the Max Planck Institute for Intelligent Systems in Tübingen in March 2018. The research was covered by University of Manchester news in 2018 and by German public radio Deutschlandfunk in July 2018.