Personally don't see any problems that cannot be solved with the right communication and tools. However, one slight downside is not getting a chance to see your team mates IRL as much as you'd like to.
When half of my former team was remote, I honestly found one of the most challenging things to be coordinating time-zones for team meetings. There are tons of apps for that but when you're working with more than 2 or 3 time-zones it becomes much more difficult!
Hmm I sometimes wonder if small remote teams aren't as close as small teams in an office, I don't know if there's value that comes from closeness though, like a positive correlation with productivity for example. But if someone craves a team they can feel close to that might be a problem they encounter on a remote team.
Language barrier can be an issue but a bit of patience solves everything. Biggest issue has been adjusting to different time zones and working around the clock, but seeing results just makes it all worthwhile!