The MBS (Multi Branch System)

Booting the E7s

Important considerations

1. The E7s should only be booted in an emergency, that is the data aquisition system is in a state from which it cannot be recovered via software.
   
2. Only reboot those E7s which show problems!
   
3. In case of a power failure, the E7s should reboot themselves automatically when power is restored.
   

Examples of situations where a reboot probably is necessary:

• no longer needed network connections refuse to disconnect even though the involved tasks have been terminated/stopped.
• some hardware fault has been detected
• SCSI bus errors are reported by the DLT drive when taping (E7_11 only)
• An E7 stops responding to input

Procedure

1. Connect the E7 to the VT100 terminal (next to the DLT drive) via the gray cable.
2. On the E7 front panel, press the "Reset/Abort" lever-switch downwards.
3. The LED-display on the E7 should change to "0"
4. Monitor the activity on the VT100 terminal: The text "Boot procedure finished" should appear, soon followed by "E7_x" (x=number of E7) in large letters and the prompt "User name:" The E7 front panel LED-display should read "8".
5. Do not log in via the VT100 terminal, but establish a telnet connection to the E7 via the X-terminal next to AXP636.
6. Proceed with rebooting the other E7s, if needed.

Back to index

Lynx System Files

Where are the files?

The files needed by the Master processor (E7_11) are located in the directory mbsrun/runxx, whereas the FRS branch Slave (E7_5) files can be found in the subdirectory mbsrun/runxx/frsbranch. If the USR branch is being used, its Slave processor (E7_6) files sit in the mbsrun/runxx/usrbranch subdirectory.

Refer to Part I of the manual for detailed information on how to prepare these files for your experiment!

NOTE: All MBS files are stored on the same file system, and most of them are freely accessible from any Lynx account!!! Files can therefore be copied, edited, and also deleted by most anybody! For this reason, take great care when you modify files! Critical files can be protected with the help of the Lynx/Unix chmod command (see some Unix documentation).

Back to index

Starting up MBS

Logging in

1. Create a terminal window on the X-terminal next to AXP636.
2. Open a telnet connection to E7_11:
   $> telnet E7_11
3. Log in as profi (no password currently required).
4. Change to the working directory:
   E7_11 profi (..) cd mbsrun/runxx

Starting the Prompter

1. Perform a "remote reset" of the whole system. For each E7, a "Reset node finished" message should eventually appear.
   E7_11 profi (..) remreset
2. Check the status of the network connections. (This should be repeated until only the message "openlog error" appears.)
   E7_11 profi (..) net600
3. Start the prompter (prm) "shell" program. Informational messages about the message logger server (E7_11) as well as clients (E7_5 and E7_6, as applicable) should appear, and the prompt should change to "E7_11>".
   E7_11 profi (..) prm
4. The prompter shell can be exited at any time by typing exit or quit, and then reentered by again typing prm. NEVER type Ctrl-z unless you want to crash the prompter or the data acquisition...

Creating the MBS Environment

1. Start the MBS data acquisition tasks (readouts, collector, transport,...):
   E7_11> @start
   A number of informational messages will appear on the screen. Watch for errors!
2. Were the CAMAC crates turned off? If so, make sure they are initialized:
   E7_11> @caminit
3. If needed, program the multi-wire detector CAMAC CFDs (thresholds/walk):
   E7_11> @cfdset

Controlling the acquisition

1. Start the data acquisition (always via the E7_5 Slave processor!):
   E7_11> e7_5::start acq
   The collector task (on E7_11) and the respective readout tasks (on E7_5, E7_6) will acknowledge that a trigger type 14 was detected and that the acquisition is running.
2. The data acquisition is stopped (always via the E7_5 Slave processor!):
   E7_11> e7_5::stop acq
   The collector task (on E7_11) and the respective readout tasks (on E7_5, E7_6) will acknowledge that a trigger type 15 was detected and that the acquisition is stopped.

Back to index

Is everything OK?

There are several useful (prompter) commands to check what is going on:

• E7_11> sho rate
  gives information about the current VME event number and how many events were processed since the acquisition was started.
• E7_11> sho rate n
  gives rate information averaged over n seconds.
• E7_11> type event -v
  types the contents of an event (including headers) on the screen. Very useful for debugging and monitoring of electronics performance.

The taping can be monitored with:

• E7_11> sho tape
  shows the tape label, what file is opened/closed, and how much data has been written to the current file and in total to the tape.

• E7_11> show task
  lists all active tasks. Useful to diagnose crashes.

Back to index

Taping

Initializing a new tape

1. Insert a new tape into the DLT drive. (The handle should only be operated when the green light is on!)
2. From the prompter, issue the initialization command. A label (max 6 letters/digits) must be provided. It is good practice to indicate the run number in the label, e.g., 'R56001'.
   E7_11> init tape label
3. The initialization takes some time–be patient. If there is an error, try again!

Mounting the tape

1. The tape is mounted with:
   E7_11> mount tape
2. and correspondingly dismounted:
   E7_11> dismount tape

Writing list-mode files

1. Open list-mode file (the only type that the MBS system can write) by
   E7_11> open file name
2. Alternatively, the command
   E7_11> @newfile
   can be used (see below).
3. Files are closed with the corresponding command
   E7_11> close file

Changing tapes

1. To change tapes, just close the current file
   E7_11> close file or @closefile (which gives info on file size and time)
2. and dismount the tape
   E7_11> dismount tape
3. When the Prompter indicates that the tape has been rewound, press the unload button on the DLT drive front panel and, when the green light goes on, operate the handle. It is good practice to write protect the used tape immediately after removing it!

Back to index

Storing magnet and feedthrough information

This can be done with the help of a special GOOSY session running under the FRS account on the (HKR cluster) VAX workstation CLARA4. The procedure to obtain a text file containing the pertinent information on magnet currents (voltages) and fields as well as target and slit drive settings/positions is as follows:

1. Connect to CLARA4, either via telnet OR by opening a DECterm window on, e.g., AXP636 and using the set host command.
2. Log in as FRS. (Ask Klaus Sümmerer for the password!).
3. Change directory to [FRS.UFC.RUNii], where ii is your experiment's run number.
4. Create the special GOOSY environment needed:
   $> @create_goosy
5. Enter GOOSY:
   $> g
6. A text file US10$ROOT:[FRS.UFC.RUNii]RUNii_Fnnn.prop containing a "snap shot" of the magnet and drive settings is created via the command
   G> @status nnn
   where nnn should be the current tape file number (or any other identifier string!).

1. In a terminal window on AXP636, log in as PROFI (if needed) and change directory to FRS$ROOT:[PROFI.GOOSY.RUNii].
2. Make sure the subdirectory FRS$ROOT:[PROFI.GOOSY.RUNii.STATUS] exists; if not, create it.
3. Execute the command file:
   $> @getstatus nnn
   where nnn is the current tape file number. The status files will be copied over into the STATUS subdirectory. The printout header will contain the filename as well as the date and time.

As a compliment to these semi-automatically produced status printouts, it is recommended to "manually" record the slit and drive positions for each tape file on so-called file sheets (a LaTeX template for preparing file sheets can be found under FRS$ROOT:[PROFI...]). Each file sheet should be accompanied by a magnet status printout from the SD program.

NOTE: Some preparatory work is needed: on CLARA4, the subdirectory US10$ROOT:[FRS.UFC.RUNii] must be created, and the files CREATE_GOOSY.COM and STATUS.GCOM must be copied over from US10$ROOT:[FRS.UFC.???] and edited. Also note that if your experiment uses "non-standard" drives, or if it involves transmitting the beam to other caves (A, B, C, or the ESR) the list of devices must be updated. Ask Klaus Sümmerer or Karl-Heinz Behr for further information.

Back to index

Executing CAMAC function calls

Back to index

Troubleshooting

Crash recovery

1. Close the current file if one is open.
   E7_11> close file
2. Stop the acquisition:
   E7_11> stop acq
3. Make a note of the error: its symptoms, causes (if known) and consequences. Forward the information to Klaus or Maggie.
4. Close any connection to on-line analysis clients and/or remote event servers. (See Connecting to MBS below for details.)
5. Attempt to stop all running tasks (except the message logger and utility task) in a controlled fashion:
   a) E7_11> e7_5::stop task -all (note the minus sign!)
   b) E7_11> e7_6::stop task -all (if the user branch is in use)
   c) E7_11> stop task -all

1. Exit the non-responding prompter with Ctrl-z.
2. Perform a remote reset:
   E7_11 profi (..) remreset
3. Check the status of all network connections:
   E7_11 profi (..) netstat

When tasks terminate abnormally, network connections can be left in an indeterminate state.The system tries to close these connections, indicated by the status (rightmost column) CLOSE_WAIT or TIME_WAIT. Just be patient! After 30-60 seconds, a repeated netstat should give only the "openlog error" message. If the status is FIN_WAIT or FIN_WAIT2, a reboot of the E7s in question might be needed, see Booting the E7s.

An alternative (for the advanced Lynx user!) is to create another connection to the E7 in question, list the active processes (with ps -axn) and then use the Lynx command kill -9 pid the offending processes. (Known as "killing the Zoombies and sending them to Hell"!)

Shutting down the system

1. Check the status of the taping: (no file open, no tape mounted):
   E7_11> show tape
   Close any open files, dismount the tape.
2. Stop the acquisition if it is running:
   E7_11> e7_5::stop acq
3. Disconnect any external analysis clients by stopping analysis processes etc.
4. Stop all tasks on the respective E7 processors:
   E7_11> @stoptasks
   (This is equivalent to issuing the stop task -all command to each E7...) If something hangs up, use Ctrl-z to exit and then perform a remote reset...
5. Log out from all E7 telnet processes:
   E7_11> logout
6. Close the X-terminal sessions.

Feedback!

Back to index

GOOSY (On-line analysis client)

General

Where are the files located?

To ensure fast I/O operations, the data base is most conveniently placed on the local disk of AXP636, the FRS-owned Alpha station in the Meßhütte, in the provided subdirectory $36$DKA100:[PROFI.DATABASE].

Back to index

Creating the data base and GOOSY environment

1. Log in as PROFI on AXP636. (Maggie recommends to use the "old" desktop setup.) Ask an FRS person for the password, unless you know it already. PLEASE: do not spread this password around to people not involved in the experiment!
2. Change to the eperiment subdirectory:
   $> set def [profi.goosy.runxx]
3. If needed, create the data base file:
   $> @create_base
   Watch carefully for error messages! If there are problems, you must locate and solve these before you go on! It is essential that the data base corresponds to the on-line analysis you want to use.
4. Start up the GOOSY environment:
   $> @init_goosy
   
5. Start GOOSY:
   $> g

Back to index

Connecting to MBS

Direct connection

1. Connect to the E7_11 event server task:
   G> @start_e711
2. Disconnect with
   G> @stop_e711

Remote event server (REVSERV)

1. Make sure the stream server task is running on the E7_11 (default as of August 1997).
   NOTE: If the event server task is running, it must first be stopped! (E7_11> stop task m_event_serv)
2. Log in as PROFI on for example AXP604.
3. Set default to the experimental account FRS$ROOT:[PROFI.GOOSY.RUNxx].
4. Start the remote event server task:
   $> @revserver
   (This will "block" the terminal window used.) A connection will be established to port 6002 (default port number for the stream server task) on E7_11. Check for messages: the remote event server port number (default: 6003) will be indicated. At the same time, the MBS prompter process should indicate that a client has been connected.
5. From the GOOSY process, connect to the remote event server via
   G> @start_mbs
   NOTE: This presupposes that the remote event server is running on AXP604 and that its port number is 6003. If this changes, update the START_MBS.GCOM file!!
6. The connection is broken with
   G> @stop_mbs
7. If the E7 need to be restarted, you must first kill the remote event server process, or the stream server task will hang up:
   a) Select the remote event server terminal window and interrupt the process with Ctrl-c
   b) Stop the interrupted process with $> stop
   NOTE: If you forget the second step, the E7_11 stream server task will hang up, causing problems with stopping and restarting the prompter...

Back to index

Is everything OK?

Back to index

Troubleshooting

Subproces $XXX disconnected...

1. G> @restart_dsp

This deletes the present DSP subprocess (if there is one) and starts a new one.
NOTE: Whenever the analysis process is restarted, the link to the E7s must be re-established (with @start_e711)!!!

Spectra are no longer being updated or behave funnily

1. Is the data acquisition on the E7s running?
2. Are valid triggers coming into the system?
3. Is the link to the MBS still open?

If the answer to any of these questions is "no", fix the problem. If it is "yes", then there is a GOOSY problem. Try the following:

1. Check the corresponding .PPL code to see if any conditions are required and, if so, whether these are fulfilled. (Most common error!)
2. Check if the relevant analysis subpackage is enabled (its B_Anal_Enable bit is '1'B) and was properly initialized (its B_Anal_Init bit is '1'B) by typing
   G> @show_anal
3. Try restarting the analysis process $ANL. (See above!)
4. Try restarting the data base manager process $DBM. (See above!)
5. In a separate terminal window, recompile the entire GOOSY analysis program (DCL command @compile_all) and restart the analysis process $ANL.

Crashburn!

1. Exit GOOSY with Ctrl-Z.
2. At the DCL prompt, type @crashburn. This will do the following:
   • Stop the GOOSY subsessions still running.
   • Dismount the data base.
   • Wait 10 seconds and then delete the data base file ($36$dka100:[profi.database]runxx.sec).
3. If any errors occur, try the manual procedure detailed in the next section.

Back to index

Shutting down the system

After all the run is finished and the calibrations are done, the following procedure should be followed in order to shut everything down in a controlled manner:

• The GOOSY session should be stopped and the data base dismounted and deleted in order to save disk space. This is conveniently done by following the instructions in the Crashburn! section. If you want to do this "manually", follow these steps:
  1. Ctrl-Z. This exits GOOSY.
  2. $> dlenv (deletes the GOOSY environment).
  3. $> mdbm dismount base name (dismounts the data base name)
  4. $> ssec (shows any 'global sections' (such as data bases) on the workstation in question) The data base you used should no longer be listed here!
  5. $> del $36$dka100:[profi.database]name.sec (deletes the data base file from the local disk). [Substitute the proper path!]
• Check that all detectors are turned off and "secured". (Note down the voltages in the log book first! They can come in handy later!) Remove all movable detectors from the beam line!
• Make a backup of the experiment files on the PROFI account—disk space on FRS$ROOT is always scarce and old user files may be deleted at any time starting one month after the end of the experiment. Contact Klaus Sümmerer if you have problems with this!
• Make a safety copy of your listmode data files—although DLT tapes are considered "safe", you should play it safe and reduce the risk of losing your precious data!
• Inform Klaus Sümmerer or Maggie Hellström of any problems or suggestions for improvement!

Back to index

Miscellaneous

Useful Lynx/Unix commands

NOTE: Lynx and Unix are case sensitive—lower case is most commonly used!!!

General

Below are listed just a few commonly used commands—for a complete description of Lynx/Unix commands, see e.g. the GSI Benutzerberatung for suitable manuals, or check with a nearby library!

• cd directory
  Sets the default directory to directory. Remember that in Unix/Lynx the syntax uses "forward slash" signs as delimiters; e.g. /frs/mbsrun/run56 .
• cd ..
  Moves up one step in the directory hierarchy.
• pwd
  Displays the path name of the current directory.
• mkdir name
  Creates a subdirectory name under the current directory.
• rmdir name
  Removes the subdirectory name. (It must be empty.)
• cat filename
  Types the contents of the file filename on the screen.
• tail -n filename
  Displays the last n lines of the (ascii) file filename on the screen.
• cp file1 file2
  Copies the contents of file1 to the file file2.
• cp file dir
  Copies the file file to the directory dir.
• mv file1 file2
  Moves the contents of file1 into the file file2 (old file2 is overwritten!).
• mv file dir
  Moves the file file to the directory dir.
• rm file
  Deletes the file file. Will ask for confirmation by displaying the file name followed by a question mark - answer "y" to proceed.
• ls
  Lists the contents of the current directory.
• ls -l
  Lists directory contents with details (dates, sizes, ...).
• kill -9 pid
  Stops the process with process identifier pid.
• ps
  Displays a list of all (sub)processes (and their pid's!).
• ps -axn
  Displays the (sub)process list with additional information. The status column is important -- watch out for "zoombie" processes ("Z")!
• man command
  Displays the manual entry (=online help) for the command command.
• grep pattern file
  Looks for the text string pattern in the file file.
• command1 | command2
  Passes the output of command1 through command2 (so-called "piping").

Back to index

Editors

The "traditional" editor vi is of course available on the Lynx platform -- if you like it and know what you are doing, please use it by all means! However, for most people who are used to the EDT editor on VMS, the SEDT (Screen EDiTor) editor available under Lynx is a better choice. It is very similar to EDIT/EDT and has extensive on-line help. It is invoked by typing e filename. The keypad diagram is shown by pressing the Help button (normally "/" on the numeric keypad).Some useful commands:

Gold .	Move to mark
Gold B	Move to end of text
Gold E	Edit new file in current buffer (asks whether to save the old one)
Gold F	Save and exit
Gold G	Insert contents of file at current position
Gold M	Set mark
Gold Q	Quit (exit without saving)
Gold S	Save current buffer in file different from output file

Back to index

Common GOOSY commands

In the following are listed some of the most common GOOSY commands used to display, integrate and fit spectra and pictures, and how to work with data elements. Refer to the official GOOSY manuals or to the GOOSY on-line help for more details!

• display spec name
  Displays the spectrum name.
  

  Qualifiers: xmin, xmax (,ymin, ymax) x-scale (2D: and y-scale) display limits; /err shows statistical error bars; /lin linear y (1D) or z (2D) scale; /log logarithmic y (1D) or z (2D) scale; bin=n binning factor; scalim=low,high defines y (1D) or z (2D) limits.

• show spec name (can be *)
  Shows information about the spectrum name.
  

  Qualifiers: /brief lists brief information; /full lists verbose information.

• display picture name
  Displays the picture name.
  

  Qualifiers: /lin linear y (1D) or z (2D) scale; /log logarithmic y (1D) or z (2D) scale.

• show picname (can be *)
  Shows information about the picture name.
• overlay name
  Overlays the spectrum name on the current display.
  

  Qualifiers: fr=n specifies a specific frame (pictures only).

• expand
  Expands an already displayed spectrum or picture.
  

  Qualifiers:xmin, xmax (,ymin, ymax) x-scale (2D: and y-scale) display limits; frame=n (could be *) specifies a specific frame (pictures only).

• display condition name
  Displays the condition window name on the current 1D spectrum.
  Qualifier:frame=n (could be *) Specifies a frame (pictures only).
• show condition counter name
  Shows number of events that fufilled condition name.
• display point
  Displays a line/cross-hair at the specified coordinates.
  Qualifiers:x (,y) desired coordinate(s), if not specified, a cursor appears; /l The "loop" qualifier.
• display polygon name
  Displays a condition polygon on current spectrum.
  

  Qualifier: frame=n (could be *) Specifies a frame (pictures only).

• integrate
  Integrates a 1D spectrum between the limits specified. (Does not work on 2D spectra!)
  Qualifiers:xmin, xmax desired limits, if not specified, a cursor appears; /l The "loop" qualifier.
• export spec name out=filename
  Exports a spectrum (or spectra!) in the standard GOOSY "spec" format (zero-suppressed ascii).
• export GD spec name out=filename
  Exports a spectrum (or spectra!) in the SATANGD format (zero-suppressed ascii).
• import spec name in=filename
  Imports a spectrum (or spectra!) into GOOSY. The spectrum must be previously defined in the data base! (GOOSY format only!)
• project name1 name2 dim=n win=min, max
  Projects the 2D spectrum name1 onto the x-axis (dim=1) or y-axis (dim=2) and puts the result into spectrum name2. If no limits are given, a cursor appears.
• fit spec name Fits the current spectrum. Requires cursor input.
  

  Qualifiers: /gauss uses a Gaussian peak shape; /backg the background will be fitted.

• replace cond win name Replaces condition window name. (1D spectra)

  Qualifier: xmin, xmax x-limits for the window.

• replace poly name
  Replaces polygon condition name (2D spectra). A dialog box appears, click "append", then use the cursor to define the polygon. Click on middle button to reobtain box, "end" saves the new polygon.
  

  Qualifier: /delete Old polygon points are deleted. Do NOT use the first time a polygon is defined!

• show mem [data]name
  Displays value of the data element name.
• set mem [data]name value
  Assigns a new value to the data element name.

In addition to these GOOSY commands, there are a number of useful "script files" (.GCOM extension). These are called from GOOSY by prefix "@". An important example is @pk "comment" which automatically plots the displayed spectrum or picture to the postscript printer (p60) in the FRS Meßhütte. The text string comment will be printed along the top border.

Back to index

Modifying the analysis code

Updating MGOOANL.EXE

1. In a terminal window (DCL), edit the relevant .PPL files and, if needed, the .TXT (global variable declarations), SET_MEM*.GCOM (data elements) and CREATE_SPEC*.GCOM (spectrum, picture and condition definitions) files.
2. Recompile and relink:
   a) $> @compile_all (if several .PPL routines were changed at the same time) OR
   b) $> @process name (if only the individual routine X$ANL_name.PPL was changed)
3. Carefully check the compiler messages–all errors must be corrected before proceeding!

NOTE: The above command files also update the text library PRIVLIB.TLB (and the object library).

Restarting the $ANL subprocess

1. Stop the input from MBS:
   G> @stop_mbs (OR @stop_e711)
2. Stop the old analysis and restart it. The latest version of MGOOANL.EXE will be used.
   G> @restart_anl
3. Discrepancies between the data base and the analysis code will result in errors during the initialization–check for error messages! If there are problems, you can revert to the previous version of the analysis code:
   a) Outside GOOSY, delete the latest version of MGOOANL.EXE.
   b) In GOOSY, restart the analysis with
   G> @restart_anl
4. If needed, clear spectra and conditions:
   G> @clear
5. Resume input from MBS:
   G> @start_mbs node port (OR @start_e711)

Back to index

Modifying the data base "on the fly"

Adding new or modifying existing spectra and pictures

1. In a terminal window, edit the file NEW_SPECTRA.GCOM: insert the spectrum and/or picture definitions. Already existing elements MUST be deleted before they can be redefined! Remember to update the corresponding CREATE_SPEC*.GCOM files!
2. Stop the input from MBS:
   G> @stop_mbs (OR @stop_e711)
3. Implement the changes:
   G> @new_spectra
4. Restart the analysis process:
   G> @restart_anl
5. Resume input from MBS:
   G>@start_mbs node port

Adding data elements

1. Make sure the text library PRIVLIB.TLB has been updated (done automatically during recompilation as described in the previous section).
2. Stop the input from MBS:
   G> @stop_mbs (OR @stop_e711)
3. Replace the data base entry corresponding to the declarations file name.TXT:
   G> @new_element name
4. Restart the analysis process:
   G> @restart_anl
5. Resume input from MBS:
   G>@start_mbs node port (OR @start_e711)

Back to index

Concluding remarks

This concludes the "On-line Guide" part of the FRS data acquisition manual. If you want more detailed information, please refer to the suggested reference litterature or contact the "experts"! If you have suggestions for improvements and/or updates, please contact Klaus Sümmerer or the author.