I noticed this problem too. It was working last week, so I'm guessing a recent update to VS Code broke something. I tried using the offline installation, but it didn't help either. Curiously running ...
When I use the UI option to turn on remote tunnel it initiates code-server from the windows host machine. However with WSL2 installed and all my code living there it would be nice if this feature ...