Mouse support isn't fully implemented in tje Chromecast version yet. The mouse does change to one when you can press a button but nothing happens when you press a button (only selecting text when double clicking).
Scrolling doesn't work either and that would make scrolling through large playlists a lot easier compared to the remote.