Linux desktop client offline through VPN

[Hug0B]

Plan

Premium

Country

France

Device

Desktop

Operating System

Ubuntu 19.04

My Question or Issue

The linux desktop client appears completely offline through my VPN (ExpressVPN)

It appears to be solely a linux client bug

it works through the web client and the same VPN

it works through the windows client (virtual machine) and the same VPN

it works through the android app and the same VPN

Both the debian package and the snap behave this way

[dlu]

I suspect that the problem is rather in your VPN/network setup. My Linux desktop client works even when I am connected to my VPN (not ExpressVPN, though).

[krisstarsanspar]

I've been having the exact same problem, also using Ubuntu 19.04 and ExpressVPN. For the record, Spotify was working fine with the VPN connected on the previous version of Ubuntu, but then when I installed Ubuntu 19.04 and Spotify on a new computer, suddenly Spotify can't find an internet connection unless I disconnect the VPN. I've tried everything that's been recommended to fix it, including a clean reinstall and changing the DNS options for my wireless connection and the proxy options in Spotify. I'm currently working in China; using a VPN is not optional. It should not be necesary to choose between checking my gmail and listening to my music.

[dlu]

I just registered for ExpressVPN and tried their manual OpenVPN config with an endpoint in my country on my Ubuntu 19.04 desktop. My spotify client still works. What vpn endpoint did you use?

[krisstarsanspar]

I usually connect to either LA or Hong Kong. I tried switching the endpoint without results. I usually avoid changing the VPN settings unless necessary, because living in China, the connection can be very finicky and changing anything means it can't get through the firewall.

[dlu]

Los Angeles 1 and Hong Kong 2 both work for me.

[krisstarsanspar]

Just tried several alternate locations (including hk2; usla1 is not available), but Spotify is still getting error code 4.

[dlu]

Do you know how to read tcpdump and/or wireshark output? I would shut down any application that uses the network, start a wireshark session and run spotify again.

[Hug0B]

Same, switching the location doesn't help. I have also just tested the ExpressVPN client on fedora 30 with the same results.

Maybe it works for you because you set up the manual openVPN configuration instead of using the ExpressVPN client ? It doesn't seem very practical to use in my case though.

[dlu]

---

Maybe it works for you because you set up the manual openVPN configuration instead of using the ExpressVPN client ?

 Maybe. I cannot check any more, as I already cancelled the test account. However, if this were the case, then you can test manual OpenVPN config as well and report the spotify issue to ExpressVPN.

[krisstarsanspar]

I tested it today with another VPN, Astrill, on the same computer, and it works. It seems to be something specific about the combination of ExpressVPN with Ubuntu 19.

[krisstarsanspar]

I installed and ran tcpdump, but have to admit that I'm at a loss as to how to read the output. What should I be looking for?

[dlu]

---

@krisstarsanspar wrote:
I installed and ran tcpdump, but have to admit that I'm at a loss as to how to read the output. What should I be looking for?

---

First you need to reduce network noise. Close all applications that might access the network. Connect to your VPN and find out which $tunnelInterface it created. It will probably be tun0. Run "tcptump -vpni $tunnelInterface". In the best case you will see nothing, i.e. no noise. Open another terminal, run "ping 8.8.8.8". tcpdump should output icmp requests and icmp replies only. Cancel the ping.

Find your home $routerIP and $routerMAC address. Find your $homeNetworkInterface, i.e. the one that connects to your router. It could be eth0, wlan0, enpXYZ or similar.

Open another terminal, run "tcpdump -vpni $homeNetworkInterface ether host $routerMAC and not host $routerIP". Run ping again. In the $tunnelInterface output you will see the icmp requests and replies again, while in the $homeNetworkInterface output you should see packets to and from your VPN server and hopefully nothing else.

Close both tcpdump processes and reopen them with one additional option each, namely "-w dumpfile.homenetwork.pcap" and "-w dumpfile.tunnel.pcap", respectively.

Start spotify and as soon as it told you that it is offline, close both tcpdump processes. Disconnect from VPN and run another tcpdump over $homeNetworkInterface saving into a third dumpfile.

You can then try to compare the VPN and nonVPN scenario. I also suggest that you install wireshark which can produce packet captures itself as well as read, filter and visualize your produced tcpdumpfiles.

[gumbeto]

I get the same thing with spotify installed as a snap on Ubuntu 19.04. I tried

spotify --show-console

and I can see a bunch of "Operation not permitted" errors on network operations. I believe this is snap containment denying some sort of permission that's required when expressvpn is connected.

[dlu]

OP said, it happens with the debian package as well, though.

---

@gumbeto wrote:

I believe this is snap containment denying some sort of permission that's required when expressvpn is connected.

---

[gumbeto]

---

@dlu wrote:

OP said, it happens with the debian package as well, though.

---

Good point, I missed that.

I just tried the deb here and it manages to connect, but crashes after a few seconds. Did the OP forget to `snap disable spotify` and was perhaps still launching the snap spotify? FWIW, it still has operation-not-permitted errors, so that may have been a red herring.

Another option that that seemed to be a workaround at first, with the snap installation:

spotify --ap=`dig +short ap.spotify.com | egrep "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | head -n1`

Spotify no longer marked me as offline and I could resume playback of what it was playing before. But it only showed cached images and couldn't play anything else 😕

[dlu]

Alright, there is indeed a problem with spotify in combination with the expressvpn client. When expressvpn connects, it reconfigures the system's DNS settings. The /etc/resolv.conf file is replaced by a symlink to /var/lib/expressvpn/resolv.conf, while the original content is in /var/lib/expressvpn/resolv.conf.orig. The snap has its own mount namespace in which /etc/resolv.conf is still a symlink to /var/lib/expressvpn/resolv.conf, where the latter does not exist inside the snap's mount namespace. A workaround could be a wrapper around expressvpn connect:

- Execute expressvpn connect

- Remove the /etc/resolv.conf symlink

- Copy /var/lib/expressvpn/resolv.conf to /etc/resolv.conf

- Start the spotify snap

I wish I saw a better way to do this, like mount options for snap, but I did not find any.

[gumbeto]

That works, great find!

Using `snappy-debug.scanlog` I could see other things that are being denied access to `/var/lib/expressvpn/resolv.conf` by AppArmor (e.g. freshclam, cups-browsed). So copying the file is probably the best.

[dlu]

I can offer a binary patch for the current version 2.2.0-1 of the expressvpn client. It won't touch the package files.

root@i3:~# systemctl stop expressvpn
root@i3:~# dpkg -l expressvpn | tail -n1
ii  expressvpn     2.2.0-1      amd64        ExpressVPN Command-line Interface
root@i3:~# debsums expressvpn
/usr/bin/expressvpn                                                           OK
/usr/bin/expressvpn-browser-helper                                            OK
/usr/lib/expressvpn/bash-completion                                           OK
/usr/lib/expressvpn/chrome/com.expressvpn.helper.json                         OK
/usr/lib/expressvpn/expressvpn-agent.desktop                                  OK
/usr/lib/expressvpn/expressvpn.init                                           OK
/usr/lib/expressvpn/expressvpn.service                                        OK
/usr/lib/expressvpn/firefox/com.expressvpn.helper.json                        OK
/usr/lib/expressvpn/icon.png                                                  OK
/usr/lib/expressvpn/libxvclient.so                                            OK
/usr/lib/expressvpn/openvpn                                                   OK
/usr/sbin/expressvpnd                                                         OK
/usr/share/doc/expressvpn/COPYRIGHT                                           OK
/usr/share/doc/expressvpn/changelog.gz                                        OK
/usr/share/man/man1/expressvpn.1                                              OK
root@i3:~# sha256sum /usr/sbin/expressvpnd
3cdbb860978b5c807844ca1b881450cff00a7ac3c75b2f263dd85bee2da3f2f6  /usr/sbin/expressvpnd
root@i3:~# base64 -d <<EOD > /tmp/expressvpn.patch
> QlNESUZGNDBmAAAAAAAAAE8AAAAAAAAA4I02AQAAAABCWmg5MUFZJlNZzdD22AAAJff3dDkgAACA
> AIAAAUpAAAFAABABCAAQABAAAAQgAFCg0aNBkBoNCh6mmnonqPKeoeTLZIW7NYgCFHRgudzyMw4f
> 1lSJlGKq+F7KPxdyRThQkM3Q9thCWmg5MUFZJlNZkYAhcgCSWEgswCIAAgBAAAQAEBj4IABUJKAN
> NpApVGgDR8FIiPdCQo7wiSolS3jkT2o23cEAkkhIK+LuSKcKEhIwBC5AQlpoORdyRThQkAAAAAA=
> EOD
root@i3:~# sha256sum /tmp/expressvpn.patch
b908d21afdfd4993b9812df94c6f00f1b0e2dd5424aee3a29e9c0ab26ca254c1  /tmp/expressvpn.patch
root@i3:~# bspatch /usr/sbin/expressvpnd /usr/local/sbin/evpnd /tmp/expressvpn.patch
root@i3:~# chmod 0755 /usr/local/sbin/evpnd 
root@i3:~# ls -l /usr/local/sbin/evpnd /usr/sbin/expressvpnd
-rwxr-xr-x 1 root root 20352480 Okt 10 21:37 /usr/local/sbin/evpnd
-rwxr-xr-x 1 root root 20352480 Jul 17 04:26 /usr/sbin/expressvpnd
root@i3:~# sha256sum /usr/local/sbin/evpnd /usr/sbin/expressvpnd
87b9dd20c7af6dd40471b7fe96a30cbb6611108c39c153a9ffdf81adbaf3c0ea  /usr/local/sbin/evpnd
3cdbb860978b5c807844ca1b881450cff00a7ac3c75b2f263dd85bee2da3f2f6  /usr/sbin/expressvpnd
root@i3:~# mkdir -p /etc/systemd/system/expressvpn.service.d
root@i3:~# cat <<EOD > /etc/systemd/system/expressvpn.service.d/use-resolv.conf-patch.conf
> [Service]
> ExecStart=
> ExecStart=/usr/local/sbin/evpnd --client-version 2.2.0 --client-build 100
> EOD
root@i3:~# systemctl daemon-reload
root@i3:~# systemctl cat expressvpn
# /etc/systemd/system/expressvpn.service

[Unit]
Description=ExpressVPN Daemon
After=network-online.target

[Service]
User=root
EnvironmentFile=-/etc/default/expressvpn
Environment=PATH=/bin:/sbin:/usr/bin:/usr/sbin
ExecStart=/usr/sbin/expressvpnd --client-version 2.2.0 --client-build 100
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/expressvpn.service.d/use-resolv.conf-patch.conf
[Service]
ExecStart=
ExecStart=/usr/local/sbin/evpnd --client-version 2.2.0 --client-build 100
root@i3:~# systemctl start expressvpn
root@i3:~# systemctl status expressvpn
● expressvpn.service - ExpressVPN Daemon
   Loaded: loaded (/etc/systemd/system/expressvpn.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/expressvpn.service.d
           └─use-resolv.conf-patch.conf
   Active: active (running) since Thu 2019-10-10 21:39:42 CEST; 3s ago
 Main PID: 8558 (evpnd)
    Tasks: 12 (limit: 4648)
   Memory: 21.3M
   CGroup: /system.slice/expressvpn.service
           └─8558 /usr/local/sbin/evpnd --client-version 2.2.0 --client-build 100

Okt 10 21:39:42 i3 systemd[1]: Started ExpressVPN Daemon.
root@i3:~# ls -l /etc/resolv.conf && cat /etc/resolv.conf
-rw-r--r-- 1 root root 19 Jan 10  2019 /etc/resolv.conf
nameserver 8.8.8.8
root@i3:~# expressvpn connect usla
Connecting to USA - Los Angeles...      100.0%                       
Connected to USA - Los Angeles

   - To check your connection status, type 'expressvpn status'.
   - If your VPN connection unexpectedly drops, internet traffic will be blocked to protect your privacy.
   - To disable Network Lock, disconnect ExpressVPN then type 'expressvpn preferences set network_lock off'.
root@i3:~# ls -l /etc/resolv.conf && cat /etc/resolv.conf
-rw-r--r-- 1 root root 65 Okt 10 21:39 /etc/resolv.conf
# Generated by expressvpn
search expressvpn
nameserver 10.73.0.1
root@i3:~# expressvpn disconnect
Disconnecting...
Disconnected.
root@i3:~# debsums expressvpn
/usr/bin/expressvpn                                                           OK
/usr/bin/expressvpn-browser-helper                                            OK
/usr/lib/expressvpn/bash-completion                                           OK
/usr/lib/expressvpn/chrome/com.expressvpn.helper.json                         OK
/usr/lib/expressvpn/expressvpn-agent.desktop                                  OK
/usr/lib/expressvpn/expressvpn.init                                           OK
/usr/lib/expressvpn/expressvpn.service                                        OK
/usr/lib/expressvpn/firefox/com.expressvpn.helper.json                        OK
/usr/lib/expressvpn/icon.png                                                  OK
/usr/lib/expressvpn/libxvclient.so                                            OK
/usr/lib/expressvpn/openvpn                                                   OK
/usr/sbin/expressvpnd                                                         OK
/usr/share/doc/expressvpn/COPYRIGHT                                           OK
/usr/share/doc/expressvpn/changelog.gz                                        OK
/usr/share/man/man1/expressvpn.1                                              OK

To get rid of it, one just needs to rm /etc/systemd/system/expressvpn.service.d/use-resolv.conf-patch.conf && systemctl daemon-reload.

[countdrunkula]

This has been driving me crazy and I have finally happened on a solution - using ExpressVPN's new lightway protocol*

You'll need the latest version of ExpressVPN installed and then on the CLI you'll need to change the protocol as follows:

$ expressvpn protocol lightway_udp

After changing that, the Spotify linux client seems to work as normal again. The caveats are that lightway is still in development  so may not be that stable, and also that it may be blocked by some restrictive firewalls as it's not using the more common tcp/443 combination that most firewalls will allow.

* https://www.expressvpn.com/blog/lightway-protocol-beta/