Spending on correction agencies in Massachusetts has exploded in the past decade despite only a modest increase in the number of people incarcerated and now accounts for a bigger chunk of the state budget than each of the departments that oversee higher education, social services, and public health, according to a new study.