HEPiX Rio Meeting ReportMinutes of the Meeting held September 24th-26th, 1995
LAFEX was one branch of CBPF and Prof Santoro presented the structure of the research establishments in Brazil together with a description of the range of physics and cosmology done at LAFEX.
When CN opened the CSF service in 1992, L3, having already ported its code to UNIX, were well placed to take immediate advantage and used a large fraction of the available resources for some time. By this time, Apollos were deemed to have no future and their gradual replacement by HP Series 700 had began. UNIX replaced Aegis on the remaining Apollos and some RS/6000s appeared. in 1994, a large SGI node was installed in SHIFT for L3 and much more UNIX capacity has since been added.
L3 had always insisted on a heterogenous environment with the result that physicists had been forced to get into contact with UNIX early and this had allowed L3 to take advantage of steadily increasing amount of CPU power at the best price/performance ratio. Since at that time LANs were bottlenecks, a central solution was necessary - such as SHIFT and similar farms. Use of CERNLIB, GEANT and standard FORTRAN eased the transition between UNIX flavours. Graphics was initially GKS-based but has since migrated to X/PHIGS and Motif.
Much of their central production control software had been written in REXX for use on the mainframe and this had needed a very large conversion effort in the move to UNIX production facilities. Code management was CMZ and Patchy and these ran everywhere. They were considering a move to CVS in 1996.
The speaker summarised the architectures as follows -
The speaker summarised saying users spend more time inside applications than with UNIX itself and a major source of irritation remained the different keyboards.
A lot of effort was being spent in smoothing the rough edges of UNIX with respect to the user environment and its differences on different architectures in the various labs. Special attention was being paid to keyboard mapping and cross-platform compatibility of tools. The speaker presented several foils of issues raised within the collaboration.
The collaboration, with SLAC Computing Services, had prepared a requirements list to guarantee institutional support. They were interested in the HEPiX Security Working Group and would be in favour of a HEPiX-wide security policy across the labs. They will consider adopting the HEPiX Login Scripts. These and other questions were being addressed in the form of a questionnaire - the so-called "Randy's List" of questions to be answered by the collaboration. Many of the issues had been or were being discussed within the HEPiX community already.
Among vendor concerns were the need for Transarc to provide timely releases of AFS for all the operating systems in common use and the need for all the vendors to support Fortran 90 properly. There was also a requirement to define a standard data interchange format for the experiment, including the selection of media.
The speaker declared that in his opinion as a physicist in BaBar, UNIX was "acceptable" but it needed improvements to its user interface and close interworking between the laboratories would be essential.
A list of requirements had been put together which included support for large file systems, hierarchical storage facilities, parallel event processing, a distributed batch queuing system and fault tolerance. To this end, several research and evaluation projects had been started, some of which had been reported at CHEP the week before.
User requirements could be summarised as single system images across a distributed environment, easier multi-vendor support, transparent access to data and file sharing (DFS was under test) and single batch queues (they were evaluating LSF).
They felt that they did not want a special HEP UNIX but a better interface to it. Security must also be improved.
Simulations had already begun (presented at CHEP). They were looking at PowerPC as the processor to use but the software question was still open - VxWorks, LynxOS, etc? Another option was to develop their own software. One important criterion was not to depend on a single vendor. Evaluation of these various options was continuing.
For the future, they were anxious to see hardware vendors adopting the latest standards and they hoped for a UNIX with better real-time features.
D0 had several hundred manyears invested in developing and using VMS utilities and there was an understandable reluctance to move away from this. He himself had experienced the adoption of UNIX in three phases:
He believed in the end that UNIX was powerful if difficult to get into and that a good Primer was missing. Turning to code porting issues, he said it had been relatively easy and only the use of non-standard features had complicated things. However, code verification against existing code was not easy, the results always varied, and not only simply rounding errors. But certain parts of D0's code suite would be very hard to part - for example they made use of the VMS search list feature and VMS debugger.
The fact of changing the computing paradigm in a running experiment (replacing dumb terminals connecting to a mainframe by workstations, X terminals and PCs) raised many debates, including whether to move directly to Windows NT.
During questions, the example of CEBAF was quoted where there had been a debate for 2 to 3 years between VMS and UNIX protaganists but where the decision had finally been taken recently to adopt UNIX.
CMS users generally preferred UNIX workstations to X terminals because there was less network disturbance, especially to graphics-related work such as PAW or data visualisation. Apart from their workstations, they made heavy use of central batch (SHIFT and CSF) and used MACs and PCs for office applications. Access to CERN from external sites was essential, including X11 access in both directions.
They had standardised the user environment on the HEPiX recommendation and were promoting the use of AFS for home directories. They had established their own X terminal boot service with some local tailoring and the standard HEPiX X environment.
Experience had shown that logins were slower than expected, probably caused by the use of the HEPiX login scripts and AFS and there was a perceived lack of user documentation on the HEPiX environment, especially how to tailor user-configurable parameters - the speaker suggested a "man HEPiX" command. And system admins reported that it was not so easy to install the present HEPiX environment. Nevertheless, the fact of having the same environment on all services was very much appreciated, and there was also praise for having a standard X11 for X terminal management.
The speaker suggested that HEPiX strive for
CMS were concerned that AFS was not an official standard and was also late in arriving for new releases of vendor operating systems. On the other hand, they were pleased to have it managed centrally with issues such as file backup taken care of for them. Token expiry was judged to be too short (set for the default of 25 hours), there was additional complication when logging in to remote nodes for long periods in cascade fashion (which token has expired?) and general AFS reliability should be improved.
The speaker acknowledged that security was important but suggested that it was mainly an issue between HEP and non-HEP sites and that inter-site working between HEP sites should be made easier.
She was in charge of some 30 nodes of which 4 were UNIX, the rest mainly VMS; most of the systems were from Digital, either VAX or Alpha. However, within her UNIX cluster there was a heterogeneous selection of systems - ULTRIX, Digital UNIX and SGI/IRIX. She made use of a common application server for the public domain software where the architecture differences were "hidden" by using a /arch directory pointed to by links from each node. The clients of the cluster were dataless (in discussions this was explained as clients which did not have local disc space except for local page and swap files).
The speaker summarised her current problems as -
CEBAF's current plans included a major acquisition to be decided shortly, with the choice falling between either a large SMP system for central processing or a new farm of processors. Important factors affecting their choice would be integer performance, price of course but also cost-effectiveness, and I/O speeds and the software environment on offer. They had a formal UNIX Procurement Committee which was in the process of reviewing the bids received.
Their next planned project would be in the area of data management and they were looking at both Storage Tek's Redwood and IBM's NTP drives, at storage silos and at HSM systems. Other current concerns included the user environment and security - could DCE be of assistance?
CERN by A.Silverman
CERN reported continuing growth in installed systems, especially HP stations and NCD X terminals. The AFS disc space was by then over 300GB spread across 7 servers but they were experiencing some problems with attached RAID controllers. On the batch side, there had been some 12 DLT units installed.
Production deployment of the HEPiX login scripts was more and more widespread and work was starting on the X11 part of the proposed HEPiX environment. The recently-installed Mail Server was being gradually brought into general use.
They were also establishing a Central Software Distribution point for the encrypted distribution of commercial software as well as a server for the provision of public domain packages, today some 40 packages with 4 to 6 architectures for each. A certification scheme for Operating Systems had been drawn up in order to maximise both consistency of the delivered product and also the use of their limited resources for support.
Over the past few months the group had been asked to perform a steady stream of software installations and were looking at ways to automate this, including the handling of operating system patches.
One of the principal new projects was Nirvana, a scheme to package various FNAL-written utilities for use by offsite groups. Among these utilities were included nedit, histoscope, the OCS (Operator Communications Software) which was in use on FNAL's production clusters and farms, and several others. Another new development in the offing was DART - UNIX-based data acquisition - for whom the first customers would go into operation next Spring.
There was central support for system installations but these were found to be labour-intensive so some automation was being sought. An AFS installation script was provided for users and the HEPiX environment was installed via the Salad procedure. They planned to migrate their X environment to the one being proposed by the HEPiX X11 Working Group.
There was a long list of planned developments:
DESY Zeuthen were still largely based on NFS with AFS only being used for applications software. There were plans to add a cartridge robot, possibly a DLT system.
SLAC were evaluating some technologies in preparation for the BaBar experiment, including SSA-connected discs on RS/6000s.
Plans were being made to migrate SPIRES to UNIX, by recoding most of it in C with the remaining parts using a 370 emulation scheme under UNIX.
NIKHEF have laid plans to reconfigure their network into a star topology using UTP with subnets. They will copy CERN's ASIS scheme for public domain software and introduce AFS.
They maintain a very active group working on audio- and video- conferencing and have established a Media On Demand service making use of many freely-available tools. They will participate in the 1996 World Internet Fair.
They were looking at an AFS/WWW project whereby URL links would benefit from the presence of AFS if available. ASIS was mirrored from CERN in the Institute of Physics and other Prague institutes were mirroring the software from there.
Development was starting on operation tools for the clusters to manage users, batch and disc space management. This would include remote management tools.
Their SP2 had arrived; it would be shared by many experiments and was attached to an IBM 3494 cartridge robot as a data repository.
Recabling of the main building was underway, installing twisted pair cables and Cisco switches with 100Mbit Ethernet as the backbone between the switches.
Among the problems under current investigation were
They had implemented an SMTP mail server and made use of Majordomo as a mail list server. There were also several information servers in operation. For tape devices on their tape servers, there was a mixture of Exabyte and DLT stackers.
Future plans around the Work Group Servers included investigating DCE (with one cell per cluster), LSF for batch work, DFS for home directories and adding POP3 support to the mail service.
The Oxford Uni Particle Physics Department was a largely Digital shop but recently some Pentiums running Windows NT had begun to make an appearance, utilising the Exceed X windows interface. There were plans to evalluate the HEPiX login scripts. Elsewhere in the Lab there was heavy use of SUNs.
She believed that auditing could be a useful, perhaps even one day essential, tool in case of troubles such a serious security incident. D-UNIX offers to build an audit demon and to provide reports. Many paramters were configurable. She had found D-UNIX enhanced security easy to use and considered it to be a good starting point but stated that it required better documentation and a more consistent user interface.
For reporting problems, they had considerd several tools, including direct alerts via pagers, Zephyr-grams and eventually they had purchased a commercial tool, TelAlert.
For system monitoring they used Patrol which checked for healthy status (for example adequate swap space, free memory, a list of required processes). They used home-grown tools for some of these checks and they had a policy of reducing priority on CPU-bound jobs on their interactive servers.
Other automated tasks included scripts to clean up the disc space periodically, removing temporary and core dump files, producing audit summaries and so on. Another important area handled by such tools was of course security and some of the tools used in security checking were described at the previous HEPiX and they had now been merged into this automated scheme.
In choosing speakers, it was hoped that experts from different labs could be used where possible but in all cases it was necessary to allocate suffucuent time for the speaker to prepare the material, a non-trivial time often. And it was also the case that the author or person responsible for a product was not always the best teacher.
The decision to have a cluster of nodes in a given WGS or PLUS rather than a single large configuration was partly in order to minimise the single point of failure risk although every cluster did have one node which had special roles (account registration and accounting for example). Nevertheless, jobs were shared across all nodes in a cluster using a tool called acrontab developed by CERN's AFS team. Similarly, incoming accesses from users were shared across the nodes by the ISS scheme of crude load balancing.
In the past months, a lot of effort had gone into understanding the system behaviour and many monitoring and analysis scripts had been developed. From the results, they could now plot the rise in use of CERNSP compared with the gradual rundown of CERNVM.
The next question was system modelling - what effect would be gained by adding extra memory for example for the same number of logged-in users? Other tools under study or in some cases in development were --
Concentration this year was on consolidation of existing clusters and services plus some help to be given to the LEP experiments to assist them to migrate off CERNVM,
The first attempt to satisfy this demand had been afsmail - storing the incoming mail spool in AFS which could then be read from any AFS client. However, use of Berkeley mail's flat file scheme for the inbox meant that the AFS file was read afresh with every incoming mail, hence continuous file transmission as the user's mail spool was updated; if the inbox was not cleared regularly this could, and did, generate a lot of AFS traffic. It did not scale.
Use of IMAP as a mail protocol was much more efficient that AFS, only the headers were transferred as they changed until the user actually requested a specific mail. CERN had therefore installed a Digital UNIX Alpha twin host server with RAID discs as an IMAP-based mail server. The configuration included several fail-safe options and automatic failover of the discs. Incoming mails were stored on the server but once saved into a user's mail folder, they were transferred to files on the user's normal home directory, usually AFS. Today the service was IMAP2-based but there were plans for IMAP4. There was also some support for POP.
Another feature of the Mail Server was that it generated so-called user-friendly addresses of the form Tony.Cass@cern.ch in the sender field automatically so that the recipient could reply to a host-independent address and the mail server would be able to deliver the reply wherever the originator accessed the server from.
A user interface package had been developed and was made independent of Majordomo, just in case the latter was later replaced by something newer and better. With this package, a list administrator could use his or her choice of editing or updating scheme, based for example on ORACLE, Filemaker Pro, a simple text editor, etc. Other tools were planned, including some using WWW.
The service had been in production for some months and many users had already successfully migrated to it, including the LHC experiments.
The goal of the migration team was now to provide a home, either on UNIX or on a PC which was as comfortable as VMS. The migration task force included representatives from the major user groups and experiments. Communication within the community was considered very important and to this end UNIX User Meetings were held fortnightly although they had not yet attracted a lot of user interest. On the other hand, the first VMS to UNIX public meeting held in August had attracted a large audience.
A VMS migration questionnaire had been issued to help identify the hot topics. People had been appointed to specific responsibilities such as mail, batch, editors, information schemes, etc. A training schedule was in preparation and they hoped to share material with other labs. The UNIX at FNAL Guide was being updated and a UNIXHelp scheme developed at Edinburgh University seemed to be interesting. All this was pulled together within the Web.
For electronic mail, they would prefer something more advanced than Pine, perhaps based on mh, for example exmh. VMS to mh folder conversion tools existed.
They expected most general VMS users to have migrated by the end of summer 1996 although some specific groups, (e.g. CMS and D0) were setting their own timetables.
Between the use of ATM and Fibrechannel switches, there was still no clear consensus. Both were featured but they had different target markets, different features, and different problems.
At CHEP the previous week to the HEPiX meeting, J.Shiers and R.Melen had run a BOF on Mass Storage where it had been agreed that
The evaluation had then switched to WABI which appeared to have fewer limitations and offered better application performance. Licensing of WABI was free on SUNs but not on some other platforms.