Recent advances in machine learning have demonstrated impressive performance on complex tasks such as human-level image understanding and natural language processing. However, the increase in size and performance of these models has been accompanied by an increase in their energy consumption. This development has led to a growing interest in sparse, energy-efficient models in recent years. In this project, we will investigate activity- and parameter sparsity in a recurrent neural network architecture. The dependence of these two types of sparsity will be studied and optimal trade-offs between performance and efficiency will be identified.