Do have the startx command installed ? You can use that from the command line without DisplayManager (Switch to a virtual console with e.g. ctrl-alt-F2). And with XFCE4 you can use the command startxfce4 to start from the command line without DisplayManager. This could help troubleshoot.