Switching perspectives with shortcuts sometimes focuses sidebar?

I noticed that when switching perspectives with shortcuts, it happens that the focus stays on the sidebar instead of the task list when the task list in that perspective didn’t get focus yet.

What is a reliable way to switch to the task list in that case? Tab doesn’t seem to work, arrow right neither.

(This also happens when the sidebar is hidden)