A new model - the factorial hidden Markov volatility (FHMV) model - is proposed for financial returns and their latent variances. It is also applicable to model directly realized variances. Volatility is modeled as a product of three components: a Markov chain driving volatility persistence, an independent discrete process capable of generating jumps in the volatility, and a predictable (data-driven) process capturing the leverage effect. An economic interpretation is attached to each one of these components. Moreover, the Markov chain and jump components allow volatility to switch abruptly between thousands of states, and the transition matrix of the model is structured in such a way as to generate a high degree of volatility persistence. In-sample results on six financial time series highlight that the FHMV process compares favorably to state-of-the-art volatility models. A forecasting experiment shows that it also outperforms its competitors when predicting volatility over time horizons ranging from one to one hundred days.