Training Language Model Agents without Modifying Language Models