We present a novel unsupervised method for face identity learning from video sequences. The method exploits Convolutional Neural Networks for face detection and face description together with a smart learning mechanism that exploits the temporal coherence of visual data in video streams. We introduce a novel feature matching solution based on Reverse Nearest Neighbour and a feature forgetting strategy that supports incremental learning with memory size control, while time progresses. It is shown that the proposed learning procedure is asymptotically stable and can be effectively applied to relevant applications like multiple face tracking and online open world face recognition from video streams. The whole system including the smart incremental learning mechanism take advantage of the GPU.