Brother, I tried everything and was about to give up on Google Home and Mini altogether...but I realized the issue ONLY happened when I was streaming from my Android (BlackBerry KeyOne). It started happening after a recent update. It was TOTALLY the notifications issue. Thanks so much for suggesting this. It made total sense as soon as I read it. Even before making the adjustment, I knew it was going to fix the issue because the notification tab has been intermittent.
Don't know if you're still having this issue, but I believe it's a notifications issue. I blocked notifications from Spotify and Google Home. Worked like a charm. Inconvenient, but works. Am sure future updates will resolve. Could also be an issue with how certain wi-fi networks handle the constant communications back and forth.