Tagging onto this since there seems to have been no followup. I'm having similar or related issues, I think.
I have OpenVMS 8.3 running on an Alpha PWS. XDM is set up with appropriate files in SYS$SPECIFIC:[TCIPIP$XDM] at least as far as I know. I'm more familiar with xdm setup on UNIX/Linux, but these look right to me. I'm on a private LAN behind a firewall, so I've just let the XDM service default to minimum security.
Using the gdm X-server on Linux (tried from both Debian and Slackware on Intel hosts) it is apparent that the broadcast is working. The host choosers do show the Alpha system as available, with a status of "Willing to Manage." Attempts to connect, however, result in just a blank screen on the X-server machine, and no further activity. I can find no suggestion on the OpenVMS logging that the connection was even noticed there, though I suppose I might not be looking in the right place.
The same gdm X-servers have no difficulty connecting to various other UNIX and Linux boxes. The problem appears only with OpenVMS. I expect it is related to a security requirement expected by OpenVMS that the Linux hosts are not meeting, but I'm not sure what to fix.
These same Linux hosts can use "ssh -Y" to get to the OpenVMS system, and will run simple DECWindows applications successfully across that link. Given the verbosity required to invoke such applications from a command line, though, and the fact that this is my own private LAN with no bandwidth issues, I'd really rather be able to use a full graphical login to the DECWindows environment.
The X logs show nothing but the usual X initialization messages.
If I ssh -Y into the OpenVMS side as SYSTEM and do @cde$path:xsession
I get some warnings about fonts not found followed by
WARNING: Authorization initialization problem.
Authentication is not available, authentication disabled.
This message appears as text in the terminal window on the originating system, and also as a pop-up window obviously formatted by DECWindows Motif. When "OK" is clicked in the popup, the mouse cursor changes to the DECWindows hourglass shape and everything freezes for about 30 seconds. Then the local system operations return to normal, though the terminal window is hung up.
I did copy the template files for XDM_CONFIG.TXT, XACCESS.TXT, and XDM_KEYS.TXT, but did not change their contents. I omitted XSERVERS.TXT since I don't want OpenVMS to try to attach any remote servers itself. I just want it to accept queries.
OK, I started from a clean reboot on the Alpha, and here's what I got.
X Client is Alpha PWS433au running OpenVMS 8.3 installed from the distribution CD. No patches beyond that have been applied, and I didn't know there was a source for official patches, so perhaps I need some.
XDM did start during the boot, as noted on the console.
I opened an XDM chooser screen on a Linux system (Wolvix, which is Slackware/SLAX 11.0) and the Alpha appeared as an option, with status of "Willing to Manage." I selected it and clicked "Connect". Screen blanked and the X logo cursor appeared. The mouse can move that around but that's it. The screen stays that way for about 5 minutes, then times out and returns to the primary local host screen.
At the time of the attempted connect, the Alpha operator console showed:
Message from user System on Asher
Event: Circuit change from: Node Routing Circuit CSMACD-0,
New Circuit State=On
After the session timed out, this was in SYS$SPECIFIC:[TCPIP$XDM]TCPIP_XDM_RUN.LOG;136
$ Set NoOn
$ VERIFY = F$VERIFY(F$TRNLNM("SYLOGIN_VERIFY"))
%TCPIP-I-INFO, waiting for DECwindows to start at 15-MAR-2009 15:17:47.18
%SYSTEM-S-NORMAL, normal successful completion
%NONAME-F-NOMSG, Message number 00000004
Contents of SYS$LOGIN:LAN$ACP.LOG;83
15-MAR-2009 15:16:51.01 Defined LAN$DLL to be SYS$SYSROOT:[MOM$SYSTEM]
15-MAR-2009 15:16:52.38 Found LAN device EWA0, hardware address 00-00-F8-76-38-78
15-MAR-2009 15:16:52.46 Node database file, LAN$NODE_DATABASE, not found
15-MAR-2009 15:16:52.67 LANACP initialization complete
Empty file for SYS$LOGIN:decw$sm.log;26
Contents of SYS$LOGIN:ACME$SERVER.LOG;83
%ACME-I-LOGOPEN, logfile opened on 15-MAR-2009 15:16:57.15
SYS$MANAGER: contains the same empty decw$sm.log;26 and a DECW$SERVER_0_ERROR.LOG;21 from last April when I had an actual graphics card in the Alpha and a local graphical console.
There is an XDM process running, started at boot up and still running after the connection attempt:
15-MAR-2009 15:57:49.66 User: TCPIP$XDM Process ID: 00000426
Node: ASHER Process name: "TCPIP$XDM_1"
User Identifier: [TCPIP$AUX,TCPIP$XDM]
Base priority: 8
Default file spec: Not available
Number of Kthreads: 1
Devices allocated: BG47:
Soft CPU Affinity: off
As for the Linux system with the X server, the XOrg and gdm logs contain only X initialization messages, identical for any startup of an X instance on that machine except for the timestamps. The system log, security log, and message log contain nothing around the time of the connection attempt.
It occurs to me that I really haven't ever set up DECNet Plus on the Alpha, since I'm not actually using it for anything. Could that be relevant? DECNet is installed, but not configured beyond whatever the installation scripts did. I don't remember ever creating a file for TCPIP that would list local host addresses either (equivalent to the /etc/hosts file on UNIX type systems.) I suppose that might be significant, though I don't see any error message suggesting it.
An attempt to start a desktop session with the command @cde$path:xsession (issued over an ssh -Y window from the same Linux server) produces one rather vague error message on the ssh text screen, "WARNING: Authentication initialization problem, Authentication is not available, authentication disabled." This same message pops up in a MOTIF style graphical window, and when the OK button on that window is clicked, the Linux box freezes for a minute or two until something times out, then returns to normal. The script in the ssh window is still waiting or in a loop, but can be freed with CTRL-Y.
Log files are similar after that attempt, except for the decw$sm.log which contains the following:
Executing mcr cde$system_defaults:[bin]dtsession
I confess, I'm stumped. Except for that authentication message, which only appears when I try to run XSESSION.COM from an ssh window, there are no apparent errors being logged. Oh, and the hardware doesn't seem to be at issue. If I boot Linux on the Alpha, then it handles XDMCP connections correctly.
I also looked at the online documentation for CDE on the HP website. Though the title page there claims to be intended for OpenVMS, the commands and directories used throughout are UNIX style and don't work on OpenVMS. I don't know whether I'm completely missing something there, or HP has posted an incorrect document.
I'd certainly welcome any ideas or insights. Thanks.
From the same "ssh -Y" session on the Linux host, logged into the Alpha, the following commands do work as they should:
(Opens a new terminal window on the Linux screen, formatted by Motif on the Alpha because the colors and borders are quite different from those provided by the active Linux windows manager.)
Other commands issued from withing the DECTERM window all seem to work as they should, displaying DECWindows applications in Motif windows on the Linux screen. Those I tried included:
$ spawn/nowait/input=nl: run sys$system:decw$clock
$ spawn/nowait/input=nl: run sys$system:decw$bookreader
$ spawn/nowait/input=nl: run sys$system:decw$notepad
$ spawn/nowait/input=nl: run sys$system:decw$puzzle
DECW$IGNORE_WORKSTATION is already set to TRUE, though it made no difference one way or another except that the message "NO GRAPHICS DEVICE FOUND" went away at the end of the console messages at boot time.
$ decw$server_transports == "DECNET,LOCAL,TCPIP"
After a reboot, a message stating that the number of console devices is zero (sorry, I don't have the exact wording, it didn't go into the log) appeared on the operator console. This is a "headless" system as far as DECWindows is concerned, because the VGA card that is present is not one recognized by OpenVMS. It works as a text-based console only. Still no success with XDMCP or CDE, and everything else continues the same.
"Starting" DECWindows on a system without a native graphics device just sets a bunch of symbols and assignments, right? There's no actual process left running? I do have an active XDM process running all the time.
I didn't think I could disable DECNet and still have functioning TCPIP. Doesn't TCPIP layer on top of DECNet somehow?
I'll hunt down all the patches and apply them, though I don't hold out much hope. I've been unable to find anyone who has succeeded in getting this to work as it should so far, though the same questions have been raised in other forums overseas.
sys$specific:[tcpip$xdm.work]altivo_20.out;1 is created but remains empty.
I assume that the reference to "protocol" in the error log refers to the X security, which ought to have been MIT-MAGIC-COOKIE-1, no? I'm always confused about which side generates this value and how the other side knows to accept it. Since there is no user logged in yet, Xauthority files seem not to be relevant, or at least, I'm not sure what file applies. But at least I can now see that an attempt is being made at a handshake and something is getting through, just not the right thing.
Just an update to this still unresolved issue. I swapped out the graphics card in the Alpha for one that is recognized by OpenVMS. That got me a full graphical login and interface on the console display, which works OK but really isn't what I want to use. I'd rather get gdm and XDMCP to work properly so I can get all the machines to interface onto one central workstation.
Still have the same issue with getting the graphical login onto a Linux box. I don't think it's a bug in gdm, since it seems to work with everything else in the world. It seems unlikely to be a bug in OpenVMS, though I found one claim in an HP forum stating that OpenVMS simply doesn't handle X logins securely. It was ambiguous whether they were referring to encryption or display access issues though.
I'll keep struggling with this until I have the answer, and when I find it, I'll put it here. It has to be a setup or configuration issue that I'm somehow missing.
Progress at last! I am writing this from Mozilla running on OpenVMS but displaying on the Linux machine's screen. I was always able to do that via ssh, but this time I logged into the full desktop via XDM. I still haven't figured out just why the problem exists, though.
I had to run the X command from a shell prompt on Linux to make it open the connection with access control completely disabled:
The Linux X display is running on :0 (vt7) simultaneously, which was the goal. Flip between the two with hotkeys. That all works once you get the session established.
Issues to be resolved:
1. I can't get the DECW$SERVER to run on VMS unless there's a graphics card in the Alpha. No card or an unrecognized card, and the server won't run. I really would prefer not to have a graphical display on the Alpha console itself, but just the text based OPA0 display. So far, the only way I've found to achieve that is by setting DECW$IGNORE_WORKSTATION to true, so I can log into the operator's terminal, and then running DECW$STARTUP.COM. It still finds the graphics display at that point and starts a server, but won't take over the display until the operator session is logged out.
2. I have to force gdm on Linux to start X for the Alpha with access control turned off. So far the only way I've achieved that is by issueing the X command manually, which is workable but inelegant.
3. After logging into VMS successfully, I still get that warning popup that says authentication encountered a problem and will not be used. I'm guessing this refers to xauth, which of course has been the root of this problem all along and is now turned off at the Linux end. But I got this message before even when it was turned on, if I tried to start a session through ssh.
Back in 2003 there were a number of posts in comp.os.vms complaining that VMS XDM did not do the magic cookie thing properly. Supposedly that has been fixed since, yet the only messages I could find from anyone who succeeded in getting a VMS login from Linux have involved disabling the access control completely. Puzzling.
It turns out that point 1 is not true. I am now running without a DECW$SERVER process on the Alpha and it seems to make no difference. XDM handles the display itself, and DECW$SERVER was not needed. This does make sense I guess, since the Alpha is not the X server in this configuration, but rather the client.
I still can't find the cause for that warning about "Authentication has encountered a problem..." but everything seems to work in spite of it.
malmberg December 12 2017 HPE only makes the most current version of OpenVMS Alpha / IA64 /VAX available to hobbyists. When I had access to the Alliance 1 program it was the same. No public downloads are allowed by HPE.
nmbonao December 08 2017 Is anybody knows OpenVMS 8.2 version downloadable version? or CD copy? Thank you very much
aarommes December 02 2017 Bitcoin and Blockchain enthusiasts ( plus distributed computing ) please connec / reply: http://www.openvmshobbyis t.com/forum/viewthread.ph p?forum_id=130&thread_id= 2991
malmberg September 10 2017 https://sourceforge.net/p /vms-ports/wiki/VMSInstal lation/ For the most part just use VMS 6.1 media instead of 7.3. But why run the older release?
DoeveR August 07 2017 Where can I find the write up on running VMS 6.1 using the simh emulator?
Bart March 20 2017 Happy to have found my password again!