Interacting with GUI from CLI (headless GUI)

I have linux server.

I expect there’s headless GUI that can be controlled by CLI for my server.

I do know it possible display GUI with XRDP. But I expect I can control it with SSH or CLI. It works fine when i’m using XRDP.

I have OpenBox (a windows manager) installed. I expect I can interact GUI with CLI or maybe there’s Python Library that can handle it.

mouseclick(2,3) # mouse click area at coordinate (2,3)

screenshot("./current_screen.png") # saving screenshot of current screen in specified path.

And another feature that library can handle it.

I found similiar library, it was pyautogui.

But pyautogui work if there’s existing GUI. I mean the python script error Display Not Found if i run it in CLI.

# t.py
import pyautogui
print(pyautogui.size())

It gave me error:

root@server-kentang:~/py# python3 t.py
Traceback (most recent call last):                           File "/usr/local/lib/python3.8/dist-packages/Xlib/support/unix_connect.py", line 76, in get_socket
    s.connect('/tmp/.X11-unix/X%d' % dno)                  FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):                           File "t.py", line 1, in <module>
    import pyautogui
  File "/usr/local/lib/python3.8/dist-packages/pyautogui/__init__.py", line 249, in <module>
    import mouseinfo
  File "/usr/local/lib/python3.8/dist-packages/mouseinfo/__init__.py", line 223, in <module>
    _display = Display(os.environ['DISPLAY'])
  File "/usr/local/lib/python3.8/dist-packages/Xlib/display.py", line 80, in __init__
    self.display = _BaseDisplay(display)
  File "/usr/local/lib/python3.8/dist-packages/Xlib/display.py", line 62, in __init__
    display.Display.__init__(*(self, ) + args, **keys)
  File "/usr/local/lib/python3.8/dist-packages/Xlib/protocol/display.py", line 58, in __init__
    self.socket = connect.get_socket(name, host, displayno)
  File "/usr/local/lib/python3.8/dist-packages/Xlib/support/connect.py", line 76, in get_socket
    return mod.get_socket(dname, host, dno)
  File "/usr/local/lib/python3.8/dist-packages/Xlib/support/unix_connect.py", line 78, in get_socket
    raise error.DisplayConnectionError(dname, str(val))
Xlib.error.DisplayConnectionError: Can't connect to display ":0": [Errno 2] No such file or directory