The rats won't leave. They have a nice place, and will continue to eat from your chicken feeders. If that fails, they will find another food source (eggs, garbage, whatever).
You need to get some anticoagulant poison, which will kill them slowly. Most rats that die from this will die in...