The Unreasonable Effectiveness of Separating the Task from the Model