Status: May 2021 - Broke it and fixed it (again) on Ubuntu 21.04, so writing up in antcipation of next time this breaks..
Who needs a GUI when you have a command line ?
I read the book, the blogs and the raves and rants and know I should automate all system admin activities with scripts, bits, hacks and band aids and control everything via SSH or HTTP/S. So I should never need to access the Ubuntu Gnome Desktop ever again. But I still do and while I can get by with ssh and scripts for many things sometimes I need the Ubuntu GUI and this is mostly required to be be accessed remotely.
It is still surprising that there is no simple standard out of box way to achieve this, rather there a number of ways to achieve this including using alternate:
- X11 Display Managers
- VNC servers
- X11 or Wayland "Compositors"
- Remote display protocols like Microsofts RDP, VNC and X11.
So like many things Linux you have an abundance of riches, but not simplicity. My needs are simple:
- Be able to access standard Ubuntu Gnome Desktop from MacOS (so need VNC)
- Get the same look and feel as if I was on local Ubuntu Desktop with a bog standard install (ie Gnome Desktop)
With Ubuntu 16.04 Remote Desktop Access as easy to get going "out of the box" but from 17 on it keeps getting broken in some way due to UI sub-systems changes. Here are my tips to help "fix it up again", hence writing this up "backwards" from getting diagnostic info to then setting up remote desktop.
Over last few Ubuntu release I have found the following things have caused x11vnc to break:
- Change from X11 to Wayland Compositor
- Shift of XAUTHORITY file from ~/.Xauthority to /run/user/<UID>/gdm/Xauthority
- Change to the X11 DISPLAY used
- Change in security management of X11 server access
So here are some tools / command I have used in getting information on what is going on.
Find all the DISPLAY being used and the owner
The X11 display is access via a unix socket, which are in directory "/tmp/.X11-unix":
--- --- Use ls of directory to find the active DISPLAYS --- # ls -la /tmp/.X11-unix/ total 16 drwxrwxrwt 2 root root 4096 May 12 13:37 . drwxrwxrwt 21 root root 12288 May 12 22:46 .. srwxrwxrwx 1 WHO WHO 0 May 12 13:37 X1 --- --- Changes 'X' -> ':' to get DISPLAY --- # ls /tmp/.X11-unix | tr 'X' ':' :1 --- --- So user 'WHO' is using display ':1' ---
Since Ubuntu 17.04 there has been seperate processes and DISPLAYs for the initial login "greeter" and the user "session". This has proved problematic for x11vnc as it means you cannot have single background process that is connected to a single display.
Also the X11 display for "greeter" is not longer available once the login has completed and you are running with the user "session".
Find the gdm and gnome processes
To find the gdm and gnome processes that are running use "sudo ps -efc" which will show:
- e - all processes
- c - including the commmand line parameters
- f - do full format listing so you get all information, including process owner
$ sudo ps -efc | grep gnome [sudo] password for WHO: gdm 53167 53149 TS 19 13:56 tty1 00:00:00 /usr/libexec/gdm-x-session dbus-run-session -- gnome-session --autostart /usr/share/gdm/greeter/autostart gdm 53268 53167 TS 19 13:56 tty1 00:00:00 dbus-run-session -- gnome-session --autostart /usr/share/gdm/greeter/autostart gdm 53270 53268 TS 19 13:56 tty1 00:00:00 /usr/libexec/gnome-session-binary --systemd --autostart /usr/share/gdm/greeter/autostart gdm 53396 53270 TS 19 13:56 tty1 00:00:06 /usr/bin/gnome-shell gdm 53468 1 TS 19 13:56 tty1 00:00:00 /usr/bin/gjs /usr/share/gnome-shell/org.gnome.Shell.Notifications gdm 53470 1 TS 19 13:56 tty1 00:00:00 /usr/libexec/at-spi2-registryd --use-gnome-session WHO 66624 62578 TS 19 15:02 pts/0 00:00:00 grep --color=auto gnome
Notice that process 53167 does the "greeter" startup.
And for gdm ...
$ sudo ps -efc | grep gdm root 1891 1 TS 19 11:01 ? 00:00:00 /usr/sbin/gdm3 gdm 1977 1 TS 19 11:01 ? 00:00:00 /lib/systemd/systemd --user gdm 1981 1977 TS 19 11:01 ? 00:00:00 (sd-pam) gdm 2043 1977 TS 30 11:01 ? 00:00:00 /usr/bin/pipewire gdm 2044 1977 TS 30 11:01 ? 00:00:00 /usr/bin/pipewire-media-session gdm 2048 1977 IDL 0 11:01 ? 00:00:00 /usr/libexec/tracker-miner-fs gdm 2061 1977 TS 19 11:01 ? 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only gdm 2074 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfsd gdm 2092 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfsd-fuse /run/user/126/gvfs -f -o big_writes gdm 2096 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfs-udisks2-volume-monitor gdm 2109 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfs-goa-volume-monitor gdm 2113 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/goa-daemon gdm 2159 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/goa-identity-service gdm 2164 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfs-afc-volume-monitor gdm 2170 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfs-gphoto2-volume-monitor gdm 2176 1977 TS 19 11:01 ? 00:00:00 /usr/libexec/gvfs-mtp-volume-monitor gdm 2465 1 TS 19 11:01 ? 00:00:00 ibus-daemon --panel disable --xim gdm 2468 2465 TS 19 11:01 ? 00:00:00 /usr/libexec/ibus-memconf gdm 2623 2465 TS 19 11:01 ? 00:00:00 /usr/libexec/ibus-engine-simple root 3260 1 TS 19 11:01 ? 00:00:04 /usr/bin/x11vnc -auth /run/user/126/gdm/Xauthority -forever -loop -noxdamage -repeat -rfbauth /home/WHO/.vnc/passwd -rfbport 5902 -shared -display :0 WHO 3261 1 TS 19 11:01 ? 00:00:04 /usr/bin/x11vnc -auth /run/user/1XXX/gdm/Xauthority -forever -loop -repeat -rfbauth /home/WHO/.vnc/passwd -no6 -rfbport 5900 -shared -display :1 root 53149 1891 TS 19 13:56 ? 00:00:00 gdm-session-worker [pam/gdm-launch-environment] gdm 53167 53149 TS 19 13:56 tty1 00:00:00 /usr/libexec/gdm-x-session dbus-run-session -- gnome-session --autostart /usr/share/gdm/greeter/autostart gdm 53172 53167 TS 19 13:56 tty1 00:00:21 /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/126/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3 gdm 53268 53167 TS 19 13:56 tty1 00:00:00 dbus-run-session -- gnome-session --autostart /usr/share/gdm/greeter/autostart gdm 53269 53268 TS 19 13:56 tty1 00:00:00 dbus-daemon --nofork --print-address 4 --session gdm 53270 53268 TS 19 13:56 tty1 00:00:00 /usr/libexec/gnome-session-binary --systemd --autostart /usr/share/gdm/greeter/autostart gdm 53311 1 TS 19 13:56 tty1 00:00:00 /usr/libexec/at-spi-bus-launcher gdm 53316 53311 TS 19 13:56 tty1 00:00:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3 gdm 53396 53270 TS 19 13:56 tty1 00:00:06 /usr/bin/gnome-shell gdm 53453 53396 TS 19 13:56 tty1 00:00:00 ibus-daemon --panel disable --xim gdm 53456 1 TS 19 13:56 tty1 00:00:00 /usr/libexec/xdg-permission-store gdm 53463 1977 TS 30 13:56 ? 00:00:00 /usr/bin/pulseaudio --daemonize=no --log-target=journal gdm 53468 1 TS 19 13:56 tty1 00:00:00 /usr/bin/gjs /usr/share/gnome-shell/org.gnome.Shell.Notifications gdm 53470 1 TS 19 13:56 tty1 00:00:00 /usr/libexec/at-spi2-registryd --use-gnome-session gdm 53485 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-sharing gdm 53487 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-wacom gdm 53488 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-color gdm 53491 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-keyboard gdm 53493 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-print-notifications gdm 53494 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-rfkill gdm 53495 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-smartcard gdm 53496 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-datetime gdm 53499 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-media-keys gdm 53504 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-screensaver-proxy gdm 53505 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-sound gdm 53507 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-a11y-settings gdm 53512 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-housekeeping gdm 53516 53270 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-power gdm 53565 1 TS 19 13:56 tty1 00:00:00 /usr/libexec/gsd-printer WHO 66998 62578 TS 19 15:04 pts/0 00:00:00 grep --color=auto gdm
Looking closely at this you can see the processes (53172) which have XAUTHORITY input, this is essential information to get access to the X11 session. So now lets get details on process which use XAUTHORITY information:
$ sudo ps -efc | grep Xauthority root 3260 1 TS 19 11:01 ? 00:00:04 /usr/bin/x11vnc -auth /run/user/126/gdm/Xauthority -forever -loop -noxdamage -repeat -rfbauth /home/WHO/.vnc/passwd -rfbport 5902 -shared -display :0 WHO 3261 1 TS 19 11:01 ? 00:00:04 /usr/bin/x11vnc -auth /run/user/1XXX/gdm/Xauthority -forever -loop -repeat -rfbauth /home/WHO/.vnc/passwd -no6 -rfbport 5900 -shared -display :1 gdm 53172 53167 TS 19 13:56 tty1 00:00:23 /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/126/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3 WHO 68297 62578 TS 19 15:11 pts/0 00:00:00 grep --color=auto Xauthority
Two of these are a result of the x11vnc startup via systemd but the other is not (53172 again) and provides details on the location of XAUTHORITY token: "/run/user/<UID>/gdm/Xauthority"
This information is needed to set up your startup script.
Checking for the XAUTHORITY tokens
Before Ubuntu 17 the XAUTHORITY tokens where written to the user's home directory. This has now changed and the tokens are written to "/run/user/<UID>/gdm/Xauthority".
To see the token and owner do "sudo ls -la":
--- --- Get users who have XAUTHORITY tokens first --- $ sudo ls -la /run/user total 0 drwxr-xr-x 4 root root 80 May 13 14:41 . drwxr-xr-x 42 root root 1260 May 13 16:38 .. drwx------ 8 WHO WHO 260 May 13 14:41 1XXX drwx------ 10 gdm gdm 320 May 13 13:57 126 --- --- And token location details --- $ sudo ls -la /run/user/126/gdm total 4 drwx--x--x 2 gdm gdm 60 May 13 11:01 . drwx------ 10 gdm gdm 320 May 13 13:57 .. -rwx------ 1 gdm gdm 98 May 13 13:56 Xauthority
UID 126 == gdm and 1XXX == "The User"
We now have information required to write systemd startup scripts.
Getting Remote Ubuntu Gnome Desktop
My needs for the remote desktop are the following:
- Have the remote desktop the same as local desktop (no difference in look / feel or tools available) (i.e. the standard Gnome Desktop)
- I do not need to be previously logged in the get access to my desktop
- I can access the remote desktop via MacOS builit in VNC (Screen Sharing) client
- I can access the remote desktop via SSH based jump host
- Must pre-authenticate before getting to login screen
This post is not about the merits of any particuler VNC server, in fact as an end user I have little interest in VNC servers. I just need remote access to my desktop on a machine that I do not have physical access to most of the time.
Each of my needs imposes particular technical requirement and to meet my needs after reviewing the helpful community guidance I decided to go with x11vnc server. This hooks into the X11 display systems and exposes this as VNC desktop on TCP/IP socket.
To allow this you will have to disable the "wayland" compositor and use the standard X11 compositor. There is not currently an Ubuntu package that provides a VNC server for a wayland compositor. There is one being developed called "wayvnc".
Disable Wayland Compositor
The first thing to do to allow remote desktop with VNC is to disable the "wayland" compositor (introduced in Ubuntu 16, I think). This is because "wayland" does not allow access to display.
To disable wayland edit: "/etc/gdm3/custom.conf"
[daemon] # Uncoment the line below to force the login screen to use Xorg #WaylandEnable=false WaylandEnable=false <==== HERE
This will result in us of good old X11 compositor.
Setup your password
As per needs (5) above, the MacOS "Screen Sharing" (VNC) client requires you to authenticate before it will provide access to the remote computer. It is also generally good practice to protect your x11vnc server from access by anyone:
--- --- Use x11vnc to setup a password file --- $ x11vnc -storepasswd Enter VNC password: Verify password: Write password to /home/WHO/.vnc/passwd? [y]/n y Password written to: /home/WHO/.vnc/passwd
With latest Ubuntu 21.04 you will find that each of the login "greeter" and user "session" gdm/X11 instances will now run under different user ids and only the user who owns the X11 DISPLAY can access it.
This means that the x11vnc process for greeter will need to run with different user id as that of the session. This also means that you will need to have passwd files for each of these users as only the owner should be able to read there own password file.
My solution to this was to copy my user generated password file to new "/home/gdm" directory and change owner go gdm:gdm and then point the login vnc process to this password file and the use session to the user's one.
This is decribed in systemd setup ...
Systemd Startup of x11vnc
To meet need (2) means I have to have remote desktop login available via VNC on reboot / startup of host.
This can be achieved by having a systemd startup script.
Since the user, DISPLAY and XATHORITY token are different for the login "greeter" and the user "session", you need either have some smart way of getting single VNC server instance to manage using the alternate DISPLAY, XAUTHORITY files or use the "hack" of having "greeter" x11vnc process on one socket and the user "session" x11vnc process on another socket.
This problem first came up in Ubuntu 18.04, but with 21.04 it has got worse as now you also have to have the x11vnc process running on the same UID as the DISPLAY owner otherwise the x11vnc process will not be able to open the display (due to permissions).
Here is my current set of systemd startup scripts to get around this (now modified to work with Ubuntu 21.04).
There "greeter" script, using the XAUTHORITY & DISPLAY information collected in "diagnotics" section above:
$ cat /lib/systemd/system/x11vnc-login.service [Unit] Description=Start x11vnc at startup. After=multi-user.target [Service] Type=simple User=gdm <<=== Ubuntu 21.04 need process DISPLAY owner id ExecStart=/usr/bin/x11vnc -auth /run/user/126/gdm/Xauthority -forever -loop -repeat -rfbauth /home/gdm/.vnc/passwd -rfbport 5902 -shared -display :0 [Install] WantedBy=multi-user.target
And the "user" session script, using information collected in the "diagnotics" section above:
$ cat /lib/systemd/system/x11vnc.service [Unit] Description=Start x11vnc at startup. After=multi-user.target [Service] Type=simple User=WHO <<=== Now the session user ExecStart=/usr/bin/x11vnc -auth /run/user/1XXX/gdm/Xauthority -forever -loop -repeat -rfbauth /home/WHO/.vnc/passwd -rfbport 5900 -shared -display :1 [Install] WantedBy=multi-user.target
As can be seen to allow you to have VNC available for login you need to have socket for login and then socket for session.
The impact of this is that to use VNC remote desktop you first connect to the login socket server (5902 in this example) and do login, after you login the screen will go black. You now close that VNC window and connect to the user session VNC server (on socket 5900 in this example) and can login to your remove desktop via VNC.
In my original posting on setting up this workaround people have come up with other alternates. However with the new security of 21.04 these are likely to break as well.
I have not tested the alternates as most rely on additional scripting which I have not had time to review.
The solution I use is rather limited as it does not allow anyone to login, just the "greeter" and the specific user in the systemd script. Ideally someone should come up with a way to allow any user to get access to their remote desktop. If you need other users to have remote acces, then this would require each user to have their one defined socket to connect to.
In my case i have exclusive access to machine, so solution works for my needs. If you have direct SSH access to machine you could possibilty avoid this by using an SSH server to setup a local application proxy that you can then use to start an x11vnc instance directly (interactively) without the need for setting up the systemd services.
In my case the server I need to access is sitting behind a firewall and is only accesible via a "jump host". So here is final step in this setup.
Accessing Remove Desktop via SSH jump host
Now for the final problem of how to get access to my remote desktop on a machine that is behind a firewall. The answer is either configure port forwarding in firewall or more generically setup a "jump host" that you can SSH into and then use this to act as VNC communications proxy (need (4)).
So the network topology looks likes a bit like this..
In this scenario there is VPN connection through firewall / VPN gateway, which allows access to SSH Jump Host, through which you can tunnel the VNC connection.
Taking away the VPN layer which is mostly transparent, we need to establish SSH tunnel through jump host. This is done via ssh client on MacOS terminal:
- "ssh -L 5904:<VNC-SERVER-HOST>:5900 <WHO>@<JUMP-HOST>" - for main session
- "ssh -L 5906:<VNC-SERVER-HOST>:5902 <WHO>@<JUMP-HOST>" for greeter session
This establishes local sockets on the MacOS VNC client machine, which are connected to SSH tunnel on <JUMP-HOST> which connect to the VNC Server socket on the <VNC-SERVER-HOST>.
ssh can do a lot of sophisticated stuff if you delve into it...
On the Mac you can now connect to the VNC server by selecting the "Go" menu and conecting to VNC server using URL:
- "vnc://localhost:5904" - for user session
- "vnc://localhost:5906" - for greeter
To to connect first invoke the greeter and then once logged in close this window and invoke use session.
This should work until the next Ubuntu upgrade ....
Be prepared for this to break with the next Ubuntu release ...
Unfortunately the x11vnc setup tends to get re-broken on a regular basis, so be prepared for your setup to fail with each new Ubuntu upgrade release.
This is why I have documented the diagnostic tools, so you can tweek your setup and get it running again.
Hopefully in the fullness of time all this will work without having to know any of this.
This is my personal blog on using x11vnc to meet my needs.
Yes, this is a hack solution based on not having time to do something better.
It would be great is there was a well documented solution that could handle arbitary users and allow access without need to be logged in and if you look at my first posting on the within Ubuntu "self help" there are various other solution variants.
I have not had the time to verify or validate these, as they mostly use shell scripts which would take some effort to understand how they work.
In fullness of time maybe wayvnc will achieve goal of providing "out of box" remote desktop, but for now the systemd work around with two x11vnc servers (one for greeter & one for session) is likely for form basis for other solution variants.
References & Links:
"Ubuntu 18.04 LTS x11vnc no longer works" - my first posting (zebity) and diagnosis of problem, now 3 years ago and still getting new updates and suggestions....
"VNC/Servers Ubuntu Community Help" - provides useful help on alternate VNC server options, but is not fully current and so you will need to find other information
wayland - what is a compositor ... "Wayland is intended as a simpler replacement for X, easier to develop and maintain. GNOME and KDE are expected to be ported to it." .
wayvnc - in the fullness of time maybe this might be an option that works ...
ssh port forwarding - you can do a lot with ssh port forwarding...