Bumblebees don't form swarms like honeybees do. But you might see a few bees flying outside a nest, especially if it's in a bird box or somewhere else above ground. These are male bees hovering outside the nest, waiting for queens to emerge so that they can mate. Male bumblebees cannot sting, and they pose no risk to you at all. Therefore, no action is needed to be taken for them. There is more information on Tree bumblebees in this article (pdf 1mb).
You can see a bumblebee drone cloud in the video below: