Figure out where they are nesting and clear that out. Be persistant and consistent. At the first sign of any nest building, get a broom, get up on a step ladder, and swipe it down. If you have a loft, get up there and look for any holes, spaces, broken windows, etc where they could possibly be getting in. If there is a particular spot where they like to hang out, you could buy "pigeon spikes" and place them there.
http://www.deterapigeon.com/ (No affiliation with this site.)
They are either hanging around to eat your bugs or eat your seeds, so making sure any feed/supplements/bird seed is carefully put away, and doing all you possibly can for pest control, will help deter them.