Convex Structure Learning in Log-Linear Models: Beyond Pairwise Potentials
Mark Schmidt, Kevin Murphy ; JMLR W&CP 9:709-716, 2010.
Abstract
Previous work has examined structure learning in log-linear models with L1-regularization, largely focusing on the case of pairwise potentials. In this work we consider the case of models with potentials of arbitrary order, but that satisfy a hierarchical constraint. We enforce the hierarchical constraint using group L1-regularization with overlapping groups, and an active set method that enforces hierarchical inclusion allows us to tractably consider the exponential number of higher-order potentials. We use a spectral projected gradient method as a sub-routine for solving the overlapping group L1-regularization problem, and make use of a sparse version of Dykstra's algorithm to compute the projection. Our experiments indicate that this model gives equal or better test set likelihood compared to previous models.
