I would agree with Art Munson with the 1/128 buffer
But I encoutered problems with some samples library, mainly tempo locked loops from some kontakt libraries that have some problems to start exactly at the right time if there isnt enough space between the start time and the midi note triggering the loops.
Mainly this doesnt happen with all DAWs, but sometimes it happens with Cubase.