Augmented reality is a technology that merges virtual reality information with the real environment in real time performance. Augmented reality applications can make use of real time recognition and tracking of objects in the environment. Deformable objects are particularly hard to track due to their complex state space. A deformable object's full state often cannot be inferred from a single frame due to self occlusions. The aim of the project is to implement an algorithm for tracking deformable objects. The tracker will combine real time RGB-D camera input with a physical simulation. Multiple RGB-D cameras will be tried. Implementation will be done in C++ and will make use of computer vision libraries. Finally, a case study will be conducted on performance of the algorithm(s) and applicability of this approach for future augmented reality applications.