Writing NML Configuration Files

Most of the options available in CMS are specified in a configuration file. There should be one line in the file for every buffer that is required and every process using CMS should have one line for every buffer it accesses. Currently there is no difference between a CMS configuration file and an NML configuration file. The files may contain only ASCII characters, and can be generated with any text editor.

Update 9-Apr-2004: See The NML Configuration Server for information on how this information can be determined at run-time and also for some extensions to this format.

Lines beginning with # or white space are comments.

Lines beginning with B specify information about a particular buffer and are called buffer lines.

Lines beginning with P specify information about how a particular process accesses a buffer and are called process lines.

Before writing the details of the buffer and process lines, you will need to review the major buffer and process types available.

Buffer and Process Types

Local processes can connect to buffers using GLOBMEM, SHMEM, FILEMEM or LOCMEM. Remote processes can connect using TCP, UDP or RPC.

GLOBMEM

GLOBMEM is intended for communication across a VME backplane between processors with minimal operating system intervention, which allows greater portability and interoperability than SHMEM, but users need to specify the addresses of memory to use and mutual exclusion is implemented with a set of locks in memory rather than with semaphores provided by the operating system. GLOBMEM may also be used with a Bit3 ISA or EISA to VME adapter.

SHMEM

SHMEM is intended for communications between tasks managed by the same operating system. The operating system allocates the memory to be shared so users do not need to specify one. Users have a choice of mutual exclusion techniques with include something similar to GLOBMEM's method, using an operating system semaphore or mutex, or disabling and enabling context switching or interrupts during the appropriate critical sections.

Mutual Exclusion for Shared Memory

One of the most significant ways of optimizing shared memory accesses is to choose a the fastest form of mutual exclusion that still allows the application to run reliably. Mutual Exclusion is used to prevent multiple tasks, threads, or processors from using the same resource at the same time. This is necessary when using shared memory to prevent a process from reading the shared memory while only part of the message has been written or vice versa. Here are the currently available options:(One of them can be placed on the buffer line after the shared memory key.)

"mutex=os_sem" This is the default. An operating system semaphore is created using the same key used for shared memory. This semaphore is taken before each read or write and released afterwards. If another process already has taken but not released the semaphore when the semaphore is taken the operating system puts the process to sleep until the semaphore is released. This is reliable, however the operating system calls to take and release the semaphore take from 100 to 500 microseconds depending on CPU and operating system to execute. This is a very long time compared to the time required to copy small messages (less than 10k) into the shared memory area.

"mutex=none" The library will make no attempt to provide mutual exclusion. This is a very dangerous option that is only recommended when some other form of mutual exclusion is implemented on top of NML. One example might be to use only a single semaphore for multiple buffers like this:

        . . . 
        rcs_sem_wait( . . .); 
        type1 = nml1->read(); // configured not to provide mutual exclusion internally 
        type2 = nml2->read(); // configured not to provide mutual exclusion internally 
        type3 = nml3->read(); // configured not to provide mutual exclusion internally 
        rcs_sem_post( . . .); 
        . . . 

"mutex=no_interrupts" If there is only a single processor running multiple tasks or threads, most operating systems will be prevented from switching tasks, or threads as long as interrupts are disabled. Since disabling and enabling interupts is often much faster than the semaphore calls, this can be an efficient means of mutual exclusion. Currently this option will only work under Lynx and VxWorks. Under Lynx it is necessary to run the process as root. You also need to know how long your system can tolerate having interrupts disabled.

"mutex=no_switching" Some operating systems allow you to disable task switching which can be used just as disabling interrupts above, except that it is a little gentler in that it only prevents other tasks from executing but allows hardware interrupts to execute. Currently this option will only work under Lynx and VxWorks. Under Lynx it is necessary to run the process as root.

"mutex=mao split" This is an experimental option. It provides mutual exclusion by splitting the buffer in half. At any given time one half is used for reading while the other half is used for writing. Only at the end of a write is a flag changed that switches which half is used for what. A small area at the beginning of the buffer is used for registering which processes are reading and which are writing at any given time. This allows many readers and one writer to access the buffer simultaneously without allowing any reader to get a partial message or require semaphore calls. However situations can occur where it is necessary to force one of the processes to wait, or return an error. When a process is forced to wait there is currently no provision for priority inheritance, which means that currently while most of the time this option is very fast compared to the others, it can occasionally take significantly longer or timeout.

FILEMEM

FILEMEM is useful primarily for debugging. Messages are read from and written to a file, or messages may be read from one file and written to another. Record locking is used to provide mutual exclusion. This is the least efficient method, however the output file can be used for post-mortem analysis and scripts can be written and used as the input file for unit testing. FILEMEM buffers must have neutral set to 1 and add "disp" to the bufferline to use the displayable method of neutral data encoding. The input file can be set by adding "in=<filename>" at the end of the bufferline or it defaults to waiting for input from stdin, the output file can be set with "out=<filename>" or it defauls to stdout.

LOCMEM

LOCMEM is useful when many modules are linked together in one thread of execution but you want to write them such that each module uses the NML API just as it would if it needed to communicate with the other modules running separately. There is no need for any mutual exclusion mechanism and memory is obtained with a simple malloc so the operating system will not exceed its limits for semaphores or shared memory segments.

TCP

TCP stands for "Transmission Control Protocol". This is the recommended method for remote connection, for most applications. It is more reliable and can handle larger messages than UDP and is more widely available than RPC.

STCP

STCP is a simplified form of the software on top of TCP. It is useful for as an interface to languages other than C++/Java where creating a binary version of the NML message structure(s) is difficult and/or there is no version of the RCS library available. It allows that applications in that language to send and receive simple text strings with low-level socket functions, while C++ and Java applications still use the normal NML API. It must be used with the display neutral encoding method. This means "disp" or "xml" must be placed on the bufferline and "xdr" or "ascii" must be removed if present.

UDP

UDP stands for "User Datagram Protocol". UDP is sometimes faster than TCP since it eliminates some of the error checking but messages should not exceed 512 bytes.

Buffer Lines

Lines declaring buffers should have the following format:

B       name    type    host    size    neut    RPC#    buffer# max_procs [Type-spec data] 

B is the literal character identifying the line as describing a buffer.

name is a string identifying the buffer.

type is a string which can currently be "SHMEM" for shared memory buffers used for communications between processes on the same CPU , "GLOBMEM" for global memory buffers on a VME or EISA backplane, "LOCMEM" for a communications between functions run within the same process (multi-threaded applications should never use LOCMEM), or "PHANTOM" for buffers that use phantom functions.

host is a the name of the host for the buffer. If remote processes need to access the buffer they will look for an NML server on this host.

size is size of the buffer in decimal.

neut should be either 0 or 1. 1 indicates that the data stored in the buffer should be formatted into a machine independent or neutral format. Remote processes can still access buffers with a O even when running on a processor using incompatible types because the NML server can encode and decode automatically. However, the buffer itself must be encoded(neut=1) if the buffer will be accessed locally by processors of incompatible types.

RPC# - A remote procedure call number.

Update 9-Apr-2004: Use of RPC is now deprecated. A TCP port or UDP port should always be specified, so the RPC# will ignored. 0 is typically used as a placeholder for this parameter.

buffer# - A server for more than one buffer will use the buffer number to determine which buffer to access for each request. Give each buffer a unique buffer number.

max_procs- The maximum number of processes allowed to use this buffer.(If the total possible connections equals n then the connection number in the process lines may be 0 to n-1.)
Update February, 2005: The limit is only enforced for local processes connected tho GLOBMEM or when mutex=mao.

The area following the max_procs number is used to store information relevant to particular types of buffers, or to set options that have defaults.

GLOBMEM buffers require a physical address. The physical address may be specified with 1 or more address equations of the form "???_addr=0xnnnnnnnn" where ??? may be a host name, process name or bus name.

SHMEM buffers expect an integer key in the type specific area. The key needs to be a unique shared memory key on your system. For many UNIX systems the command "ipcs" can tell you which shared memory keys are in use. All of the NML buffers using SHMEM on a particular host need to have different keys.

To configure the use of TCP or UDP instead of RPC for remote access to the buffer, add either "TCP=" or "UDP=" and a port number to the buffer line. Use a port number greater than 1024 since ports 1-1023 are reserved for processes with user ID 0. It is recommended that you select a port between 5000 and 32000 to minimize the chance of conflicting with another application.

To use an alternative neutral encoding scheme where the messages are converted to ascii or plain text strings, add the word "ascii" to the buffer line. Or add "disp" to use an neutral encoding scheme that produces text strings with look much better for display. (The "disp" is required for FILEMEM.)

Update 9-Apr-2004: Use of "ascii" encoding is deprecated. For compatibility with previous versions of NML "xdr" should no be explicitly placed on the bufferline. A warning may be printed if no method is explicitly mentioned. XDR stands for "External Data Representation" a standard used by RPC. Two new encoding options available are "packed" and "xml". See NML support for XML for information concerning xml. "packed" encoding is unique to NML, it is binary similiar to XDR but not compaible with it. It packs structures more tightly and avoids byte-endian conversions when not necessary by adding the original endian type to the message. "packed" is the new default.

Update 14-Jun-2004: "packedl64" can be used to store long and unsigned long types in 64 bits. This is only useful if used on a system that uses 64 bit longs and unsigned longs, such as Itanium systems. (See NML 64bit long notes.)

To set the input file for FILEMEM add "in=" and the filename to the buffer line. Without this FILEMEM will use stdin.

To set the output file for FILEMEM add "out=" and the filename to the buffer line. Without this FILEMEM will use stdout.

To force FILEMEM to limit the number of messages in the output file add "max_out=" and the maximum number of messages to store in the output file. When this limit is reached the file will be overwritten starting at the beginning.

To enable queuing of messages in the buffer, add the word "queue" to the buffer line. The size of the buffer determines how many messages can be simultaneously queued. Ie a 10k buffer might have 2 5k messages, 10 1k messages or even a combo like 1 5k message and 5 1k messages. The server allocates several temperary buffers that are only ever used for 1 message at a time. If the buffer size is much larger than the size of the largest message adding "max_message_size=" and a smaller size may reduce the amount of memory needed by the server without reducing the number of small messages that could be stored. Updated 29-Aug-2008

To enable supplemental timing diagnostic information to be logged to the buffer, add the word "diag" to the buffer line.( See Supplementary NML Timing Diagnostics Tools.)

To create a second semaphore needed for blocking_reads, add "bsem=<key>" where key is a unique semaphore key on that host. The key should also be different thatn the shmem key(s).

To force backwards version compatibility, add "version=<version number>".

To force the server to send confirmation messages back to a remote writer add "confirm_write". This allows a remote writer to detect problems such as a full queue or if a write_if_read failed because the buffer was not read but it reduces performance.

To set the baudrate to use on an RS232 serial port add "baud_rate=n" where n is and integer baud rate. The baud rates available depend on your hardware and operating system. The default is 9600. This parameter has no effect on non-serial port communications.

To prevent GLOBMEM from using "taskLock()/taskUnlock()" under VxWorks add "NO_TASK_LOCK". Note: the taskLock() is needed when multiple tasks on the same processor access the same GLOBMEM buffer, to ensure deadlocks do not occur. This tag can also be placed on the process line to effect only that process rather than all processes accessing this buffer.

Update February-2005: To select the use of IPV6 addresses and protocols instead of the default IPV4 addressess and protocols for remote TCP,STCP and UDP connections add "use_ipv6" . Warning: IPV6 addresses beginning with hexidecimal digits fe8,fe9,fea, or feb are link-local addresses. Many systems default to only having a link-local IPV6 address. These require binding to specific interfaces and therefore currently do not work with NML.

Update March-2006: To indicate that a local clone buffer will be used on a given host add "cloned_on=<hostname/ip_address>" process on this host will read from a local shared memory copy of the buffer rather than to original remote buffer. One process on this hostshould have "do_cloning" set on the process line. Also a blocking semaphore is needed since the cloning uses a blocking_read.

Update February-2009: To set the name of the format function that should be used with the buffer use "format_name=" . The format name should not include the final "_format" If the NML constructor should have been passed ex_format as the first argument, the tag in the nml file "format_name=ex" would accept this. If the format name does not match the process will print an error message, NML::valid() will return false, and NML::error_type will be set to NML_FORMAT_NAME_DOES_NOT_MATCH_ERROR. "format=" is also accepted as a shorter verion of "format_name=".

Update February-2009: To set the name of the header file that defines the types of messages used with this buffer. Add "header=". It can be used by some tools including the Java diagnostics tool, and can serve as additional documentation. If format_name is not set then it will check for a format name that is a substring of the header name.

Update February-2009: To set the name of the C++ source file that includes the automatically generated format and update function implementations add "format_source=" It can be used by some tools including the Java diagnostics tool, and can serve as additional documentation. If format_name is not set then it will check for a format name that is a substring of the format source file name.

Process Lines

Process lines should have the following format:

P      name    buffer    type    host    ops    server timeout master c_num 

P is the literal character identifying the line as describing a process.

name is a string matching the process name passed to the CMS config function or the NML constructor. Update 2005.Februaryif the name is given is "default" then the information will be used by any process without an explicit process line already given.

buffer is the string matching the identifier used in a buffer line to which this process intends to connect. Update 2005-February: if the name is given is "default" then the information will be used by the given process for any buffer without an explicit process line already given. Having default set for both the process name and buffer name matches all combinations.

type is a string which can currently be "LOCAL" , "REMOTE","AUTO". LOCAL processes normally connect directly to the buffer via shared memory regardless of the buffer or process hosts. REMOTE process always use UDP or TCP to connect even when running on the same machine. Update 2005-February: AUTO processes attempt to determine if a LOCAL direct shared memory connection is possible and to use it if it appears to be and connect via the REMOTE method otherwise. Servers should always be LOCAL.

host is the name of the processor that the process is expected to run on. Update February-2005: This is not used unless bind_proc_host is added to the end of the process line.

ops is a string indicating whether the process will read or write. R, W or RW

server should be either 0, 1, 2, or 3. A 1 indicates that this process is responsible for running a server for the buffer, a 2 indicates that the nml_start function should spawn a server for the buffer, and a 0 is for processes that will neither spawn nor run a server for the buffer.

Update 9-Apr-2004: 3 indicates that a server should be spawned immediately rather than waiting for nml_start to be called. Unlike the other options, the TCP port or UDP port can not be shared with any other buffers.

timeout is the time in seconds for the process to wait for the mutual exclusion semaphore or for an RPC call to return before an error is reported. A time can be specified in floating point and the timeout is measured as close as possible for the given platform and process type or "INF" to never timeout.

master is a flag which should be 0 or 1, to indicate whether this process should create and initialize this buffer. On some operating systems there must be one and only one master process, on some there can be multiple masters. The parameter is ignored for REMOTE processes. There must be atleast one master that is a local process started before non-master and remote processes are started.

Update 9-Apr-2004: On some operating systems even non-masters can be started first and they will create the buffer, making this parameter almost irrelavent. We may attempt to eliminate this behavior in the future. For maximum portability, setting only a single process to be the master and ensuring it is started first is still recommended.

c_num is the connection number for this process.c_num must be between 0 and n-1 where n is the number of total_connections specified on the buffer line. Update February-2005: This is not used unless the BufferType is GLOBMEM or mutex=mao is specified and autocnum is not specified.

The area of the process line after the connection number is used to choose some options that have defaults. These options can be added in any order.

To have a remote processes set the non-blocking flag on its socket, add "poll". A remote process with the poll flag set will not expect a remote server to reply when read or peek is called, but rather to get that data in some future call to read or peek so data is generally delayed by one or more cycles but less time is spent waiting for the server.

To have a remote process setup a subscription with a TCP server add, "sub=<subscription interval>". The subscription interval is given in seconds as a floating point number. Example: To setup subscription interval of 20 milliseconds add "sub=0.02".

To prevent GLOBMEM from using "taskLock()/taskUnlock()" under VxWorks add "NO_TASK_LOCK". Note: the taskLock() is needed when multiple tasks on the same processor access the same GLOBMEM buffer, to ensure deadlocks do not occur. This tag can also be placed on the buffer line to effect all processes accessing this buffer.

To have a process use an RS232 serial port for communications add "serialPortDevName=<name>" where name is a device name such as "/dev/ttyb" or "COM2:". In addition the process must either be a server or be REMOTE. This is available only on certain platforms. By default, the port will be set to 9600 baud, 8 data bits, 1 stop bit, no parity.

Update February-2005: "bind_proc_host" forces the process to bind to the specific address in the host field on the process line. It is necessary for some IPv6 addresses on some platforms where binding to "any" address is not allowed.

Update March-2006: To have a remote process ignore a connection failure add "ignore_connect_err" allowing another alternative way of having a remote process start earlier than the server it needs. The process will attempt to make another connection each time a function that needs one such as read or write is called until a successful connection is made.

Update March-2006: To skip the usual checking for a remote process to verify that buffer name matches the buffer name known to server add "no_verify_buf" and perhaps speed up the startup process.

Update March-2006: "read_timeout=", "write_timeout=",or "connect_timeout=" override the previously set timeout but only for read/peek operations, or write operations, or when connecting.

Update March-2006: To make this process responsible for creating a local clone buffer and for spawning a thread to maintain it by copying data from this remote buffer add "do_cloning". The local clone buffer is a buffer that will be available on this host for other processes that need the same data.

Update February-2009: "retry=" sets a UDP retry period. To have an effect it must be on the client process line of a remote client on a buffer accessed via UDP. Value set should be a period in seconds for a response from the server to wait before automatically resending a request.

Update July-2009: "do_not_print_errors" prevents errors from this particular buffer and process from being printed. It could be used if known/unimportant errors from this buffer are making it potentially more difficult to seee errors related to other buffers.

Update July-2009: "do_not_print_timeout_errors" prevents errors from this particular buffer and process from being printed if they are related to a timeout. On some systems timeouts are common. It could be used if known/unimportant errors from this buffer are making it potentially more difficult to seee errors related to other buffers.

Example: NML Configuration File

This configuration file provides a possible configuration file for the rest of the examples in this document. On your system, at the very least the host names will have to be changed. Although ex_buf4 is never used by any of the examples, but it is included as an example of the use of GLOBMEM.(The tab-stops on your browser or text editor may not make the columns line up neatly. NML only needs "some" white space between the items so feel free to add/remove tabs and spaces as needed for better appearance.)

ex_cfg.nml

# ex_cfg.nml  # buffers: # nametypehost        sizeneut RPC# buffer# max_proc [type-spec] #GLOBMEMhost        sizeneut RPC# buffer# max_proc phys_addr #SHMEMhost        sizeneut RPC# buffer# max_proc key  B ex_buf1SHMEMlocalhost5120    0 1 * 101 TCP=5001 format_name=ex packed B ex_buf2SHMEMlocalhost5120    0 2 * 102 TCP=5001 format_name=ex packed B ex_buf3GLOBMEMvx405120    0 3 * vme_addr=0x4e00000 format_name=ex packed  # processes: # namebuffertypehost opsserver timeoutmaster c_num P ex1_procex_buf1LOCALmypc W0INF 10 P ex2_procex_buf1LOCALmypc W0INF 11 P ex2_procex_buf2LOCALmypc W0INF 11 P ex3_procex_buf1LOCALmypc R0INF 12 P ex4_procex_buf1LOCALmypc W0INF 03 P ex5_procex_buf1PHANTOM192.168.1.1 W00.5 14 P ex6_procex_buf1LOCAL192.168.1.1 RW00.5 15 P ex8_procex_buf1REMOTErosie RW010.0 06 P ex8_svrex_buf1LOCAL192.168.1.1 RW1 5.0 18 P ex8_svrex_buf2LOCAL192.168.1.1 RW1 5.0 18 P ex9_svrex_buf1LOCAL192.168.1.1 RW1 5.0 19 P ex9_svrex_buf2LOCAL192.168.1.1 RW1 5.0 19 P ex10_svrex_buf1LOCAL192.168.1.1 RW1 5.0 110 P ex10_svrex_buf2LOCAL192.168.1.1 RW1 5.0 110  # special default process line that is matched by anybody looking for a process # line and is not explicitly listed above. P default       default AUTOanyhost RW0 5.0 011 

---

Last Modified: February-2009

If you have questions or comments regarding this page or you would like to be notified of changes to the RCS library via email, please contact Will Shackleford at shackle [at] cme.nist.gov (shackle[at]cme[dot]nist[dot]gov)