Our contribution is twofold. First, we study model selection approaches for discriminative models, in particular for CRFs and propose to penalize the CRFs with the elastic net. Since the penalty term is not differentiable in zero, we consider coordinate-wise optimization. The comparison with the performances of other methods demonstrates competitiveness of the CRFs penalized by the elastic net. Second, we show that the proposed approach is able to take profit of the sparsity to speed up processing and hence potentially handle very large dimensional models.