Common smartphone authentication mechanisms (e.g., PINs, graphical passwords, and fingerprint scans) are not designed to offer security post-login. Multi-modal continuous authentication addresses this issue by frequently and unobtrusively authenticating the user via behavioral biometric signals, such as touchscreen interaction and hand movements. Because smartphones can easily fall into the hands of the adversary, it is critical that the behavioral biometric information collected and processed on these devices is secured. This can be done by offloading encrypted template information to a remote server, and then performing authentication via privacy-preserving protocols. In this paper, we demonstrate that the energy overhead of current privacy-preserving protocols for continuous authentication is unsustainable on smartphones. To reduce energy consumption, we design a technique that leverages characteristics unique to the authentication setting in order to securely outsource computation to an untrusted Cloud. Our approach is secure against a colluding smartphone and Cloud, thus making it well suited for authentication. We performed extensive experimental evaluation. With our technique, the energy requirement for running an authentication instance that computes Manhattan distance is 0.2 mWh, which corresponds to a negligible fraction of the smartphone's battery capacity. In addition, for Manhattan distance, our protocol runs in 0.72 and 2 s for 8 and 28 biometric features, respectively. We were also able to compute Hamming distance in 3.29 s, compared with 95.57 s achieved with the previous fastest outsourced computation protocol (Whitewash). These results demonstrate that ours is presently the only technique suitable for low-latency continuous authentication (e.g., with authentication scan windows of 60 s or shorter).