qsub(1)
qsub - submit an NQS batch request.
SYNOPSIS
qsub [ flags ] [ script-file ]
DESCRIPTION
Qsub submits a batch request to the Network Queueing System
(NQS).
If no script-file is specified, then the set of commands to
be executed as a batch request is taken directly from the
standard input file (stdin). In all cases however, the
script file is spooled, so that later changes will not
affect previously queued batch requests.
All of the flags that can be specified on the command line
can also be specified within the first comment block inside
the batch request script file as embedded default flags.
Such flags appearing in the batch request script file set
default characteristics for the batch request. If the same
flag is specified on the command line, then the command line
flag (and any associated value) takes precedence over the
embedded flag. See the section entitled: LONG DESCRIPTION
for more information on embedded default flags.
What follows is a terse definition of the flags implemented
by the Qsub command (see the section: LONG DESCRIPTION for
the complete definition and syntax used for each of these
flags).
-a - run request after stated time
-bb - broadcast message when the request begins
execution
-be - broadcast message when the request ends
execution
-d - delete the script file after spooled
-e - direct stderr output to stated destination
-eo - direct stderr output to the stdout destination
-h - print short synopsis of switches
-ke - keep stderr output on the execution machine
-ko - keep stdout output on the execution machine
-lc - establish per-process corefile size limit
-ld - establish per-process data-segment size limits
-lf - establish per-process permanent-file size limits
-lF - establish per-request permanent-file space limits
-lm - establish per-process memory size limits
-lM - establish per-request memory space limits
-ln - establish per-process nice execution value limit
-ls - establish per-process stack-segment size limits
-lt - establish per-process CPU time limits
-lT - establish per-request CPU time limits
-lV - establish per-request temporary-file space limits
-lw - establish per-process working set limit
-mb - send mail when the request begins execution
-me - send mail when the request ends execution
-mr - send mail when the request is restarted
-mt - send mail when the request is transferred to the
execution machine
-mu - send mail for the request to the stated user
-nr - declare that batch request is not restartable
-o - direct stdout output to the stated destination
-p - specify intra-queue request priority
-q - queue request in the stated queue
-r - assign stated request name to the request
-re - remotely access the stderr output file
-ro - remotely access the stdout output file
-rs - specify that a request is restartable
-s - specify shell to interpret the batch request
script
-v - print the current Qsub version number
-x - export all environment variables with request
-z - submit the request silently
LONG DESCRIPTION
As described above, it is possible to specify default flags
within the batch request script file that configure the
default behavior of the batch request. The algorithm used
to scan for such embedded default flags within an NQS batch
request script file is as follows:
1. Read the first line of the script file.
2. If the current line contains only whitespace
characters, or the first non-whitespace character
of the line is ":", then goto step 7.
3. If the first non-whitespace character of the
current line is not a "#" character, then goto
step 8.
4. Skip over whitespace until the next token is
reached. If the token is not "QSUB" and not "@$"
, then goto step 7.
5. If no "-" is present as the first non-whitespace
character following the "@$" sequence, then goto
step 8.
6. Process the embedded flag, stopping the parsing
process upon reaching the end of the line, or upon
reaching the first unquoted "#" character.
8. End. No more embedded flags will be recognized.
Here is an example of the use of embedded flags within the
script file.
#
# Batch request script example:
#
# QSUB -a "11:30pm EDT" -lt "21:10, 20:00"
# # Run request after 11:30 EDT by default,
# # and set a maximum per-process CPU time
# # limit of 21 minutes and ten seconds.
# # Send a warning signal when any process
# # of the running batch request consumes
# # more than 20 minutes of CPU time.
# QSUB -lT 1:45:00
# # Set a maximum per-request CPU time limit
# # of one hour, and 45 minutes. (The
# # implementation of CPU time limits is
# # completely dependent upon the UNIX
# # implementation at the execution
# # machine.)
# QSUB -mb -me # Send mail at beginning and end of
# # request execution.
# QSUB -q batch1 # Queue request to queue: batch1 by
# # default.
# QSUB # No more embedded flags.
#
make all
The following paragraphs give the detailed descriptions of
the flags supported by the Qsub command.
-a date-time
Do not run the batch request before the
specified date and/or time. If a date-time
specification is comprised of two or more tokens
separated by whitespace characters, then the
date-time specification must be placed within
double quotes as in: -a "July, 4, 2026 12:31-
EDT", or otherwise escaped such that Qsub and
the shell will interpret the entire date-time
specification as a single character string.
This restriction also applies when an embedded
default -a flag with accompanying date-time
specification appears within the batch request
script file.
is relatively flexible. Unspecified date and
time values default to an appropriate value
(e.g. if no date is specified, then the current
month, day, and year are assumed).
A date may be specified as a month and day
(current year assumed), or the year can also be
explicitly specified. It is also possible to
specify the date as a weekday name (e.g.
"Tues"), or as one of the strings: "today", or
"tomorrow". Weekday names and month names can
be abbreviated by any three character (or
longer) prefix of the actual name. An optional
period can follow an abbreviated month or day
name.
Time of day specifications can be given using a
twenty-four hour clock, or "am" and "pm"
specifications may be used alternatively. In
the absence of a meridian specification, a
twenty-four hour clock is assumed.
It should be noted that the time of day
specification is interpreted using the precise
meridian definitions whereby "12am" refers to
the twenty-four hour clock time of 0:00:00,
"12m" refers to noon, and "12-pm" refers to
24:00:00. Alternatively, the phrases "midnight"
and "noon" are accepted as time of day
specifications, where "midnight" refers to the
time of 24:00:00.
A timezone may also appear at any point in the
date-time specification. Thus, it is legal to
say: "April 1, 1987 13:01-PDT". In the absence
of a timezone specification, the local timezone
is assumed, with daylight savings time being
inferred when appropriate, based on the date
specified.
All alphabetic comparisons are performed in a
case insensitive fashion such that both "WeD"
and "weD" refer to the day of Wednesday.
Some valid date-time examples are:
01-Jan-1986 12am, PDT
Tuesday, 23:00:00
11pm tues.
tomorrow 23-MST
execution. This message will be delivered to all
terminals the user has within the NQS domain.
It includes such information as the request
sequence number, the request name, the execution
host and the time. The message is broadcast to
the list of hosts in the NQS domain. The
default list is in /usr/lib/nqs/nqs-domain.
This list can be overridden by a list in a file
in the user's home directory on the execution
machine called .nqs-domain. This file must have
the same format as the system default file, but
may contain different system names.
-be Broadcast a message when the request ends
execution. This message will be delivered to
all terminals the user has within the NQS
domain. It includes such information as the
request sequence number, the request name, the
execution host, and the time. The message is
broadcast to the list of hosts in the NQS
domain. The default list is in
/usr/lib/nqs/nqs-domain. This list can be
overridden by a list in a file in the user's
home directory on the execution machine called
.nqs-domain. This file must have the same
format as the system default file, but may
contain different system names.
-d If the script file is successfully spooled,
delete the original copy specified on the
command line.
-e [machine:][[/]path/] stderr-filename
Direct output generated by the batch request
which is sent to the stderr file to the named
[machine:][[/]path/] stderr-filename.
The brackets "[" and "]" enclose optional
portions of the stderr destination machine,
path, and stderr-filename.
If no explicit machine destination is specified,
then the destination machine defaults to the
machine that originated the batch request, or to
the machine that will eventually run the
request, depending on the respective absence, or
presence of the -ke flag.
If no machine destination is specified, and the
path/filename does not begin with a "/", then
the current working directory is prepended to
that the -ke (keep stderr) flag is also absent.
In all other cases, any partial path/filename is
interpreted relative to the user's home
directory on the stderr destination machine.
This flag cannot be specified when the -eo flag
option is also present.
If the -eo and -e [machine:][[/]path/] stderr-
filename flag options are not present, then all
stderr output for the batch request is sent to
the file whose name consists of the first seven
characters of the request-name followed by the
characters: ".e", followed by the request
sequence number portion of the request-id
discussed below. In the absence of the -ke
flag, this default stderr output file will be
placed on the machine that originated the batch
request in the current working directory, as
defined when the batch request was first
submitted. Otherwise, the file will be placed
in the user's home directory on the execution
machine.
If the stderr-filename contains any #
characters, they will be replaced by the numeric
part of the request's sequence number. This
provides extra flexibility in naming the stderr
output file. The # character is ignored in all
other parts of the file path.
-eo Direct all output that would normally be sent to
the stderr file to the stdout file for the batch
request. This flag cannot be specified when the
-e [machine:][[/]path/] stderr-filename flag
option is also present.
-h Print a short summary of the various Qsub
switches. The -h switch is ignored when found
within a script.
-ke In the absence of an explicit machine
destination for the stderr file produced by a
batch request, the machine destination chosen
for the stderr output file is the machine that
originated the batch request. In some cases
however, this behavior may be undesirable, and
so the -ke flag can be specified which instructs
NQS to leave any stderr output file produced by
the request on the machine that actually
executed the batch request.
specified, and cannot be specified if an
explicit machine destination is given for the
stderr parameter of the -e flag.
-ko In the absence of an explicit machine
destination for the stdout file produced by a
batch request, the machine destination chosen
for the stdout output file is the machine that
originated the batch request. In some cases
however, this behavior may be undesirable, and
so the -ko flag can be specified which instructs
NQS to leave any stdout output file produced by
the request on the machine that actually
executed the batch request.
This flag cannot be specified if an explicit
machine destination is given for the stdout
parameter of the -o flag.
-lc per-process corefile size limit
Set a per-process maximum core file size limit
for all processes that constitute the running
batch request. If any process comprising the
running request attempts to exit creating a core
file whose size would exceed the maximum per-
process core file size limit for the request,
then the core file image of the aborting process
will be reduced to the necessary size by an
algorithm dependent upon the underlying UNIX
implementation.
Not all UNIX implementations support per-process
corefile size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process corefile size
limit.
-ld per-process data-segment size limit [ , warn-limit ]
Set a per-process maximum and an optional
warning data-segment size limit for all
processes that constitute the running batch
request. If any process comprising the running
request exceeds the maximum per-process data-
segment size-limit for the request, then that
underlying UNIX implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-process data-segment warning size
limits. When a warning limit is exceeded, a
signal as determined by the underlying UNIX
implementation is delivered to the offending
process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -ld flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-process
data-segment size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process data-segment
size limit.
-lf per-process permanent-file size limit [ , warn-limit ]
Set a per-process maximum and an optional
warning permanent-file size limit for all
processes that constitute the running batch
request. If any process comprising the running
request attempts to write to a permanent file
such that the file size would increase beyond
the maximum per-process permanent-file size
limit for the request, then that process is
terminated by a signal chosen by the underlying
UNIX implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-process warning permanent-file size
limits. When a warning limit is exceeded, a
implementation is delivered to the offending
process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lf flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-process
permanent-file size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
At the time of this writing, the author was
unaware of any UNIX implementation that made a
distinction at the kernel level, between
permanent, and temporary files. While it is
certainly possible to construct a pseudo-
temporary file by first creating it, and then
unlinking its pathname, the disk space allocated
for such a file will be allocated from the same
pool of disk space that all other UNIX files are
allocated from. Furthermore, such a file will
be subject to the same quota enforcement
mechanisms, if any are provided by the
underlying UNIX implementation, that all other
UNIX files are created under.
For all UNIX implementations that do not support
a distinction between permanent, and temporary
files at the kernel level, this limit is
interpreted as a per-process file size limit,
with the word permanent removed from the
definition.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process permanent-file
size limit.
-lF per-request permanent-file space limit [ , warn-limit ]
warning cumulative permanent-file space limit
for all processes that constitute the running
batch request. If any process comprising the
running request attempts to write to a permanent
file such that the file space consumed by all
permanent files opened for writing by all of the
processes in the batch request, would increase
beyond the maximum per-request permanent-file
space limit for the request, then all of the
processes in the request will be terminated by a
signal chosen by the underlying UNIX
implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-request warning permanent-file space
limits. When such a warning limit is exceeded,
a signal is delivered to one or more of the
processes comprising the running request,
depending upon the underlying UNIX
implementation.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lF flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-request
permanent-file space limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
At the time of this writing, the author was
unaware of any UNIX implementation that made a
distinction at the kernel level, between
permanent, and temporary files. While it is
certainly possible to construct a pseudo-
temporary file by first creating it, and then
unlinking its pathname, the disk space allocated
for such a file will be allocated from the same
pool of disk space that all other UNIX files are
allocated from. Furthermore, such a file will
mechanisms, if any are provided by the
underlying UNIX implementation, that all other
UNIX files are created under.
For all UNIX implementations that do not support
a distinction between permanent, and temporary
files at the kernel level, this limit is
interpreted as a per-request file space limit,
with the word permanent removed from the
definition.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-request permanent-file
space limit.
-lm per-process memory size limit [ , warn-limit ]
Set a per-process maximum and an optional
warning memory size limit for all processes that
constitute the running batch request. If any
process comprising the running request exceeds
the maximum per-process memory size limit for
the request, then that process is terminated by
a signal chosen by the underlying UNIX
implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-process warning memory size limits.
When a warning limit is exceeded, a signal as
determined by the underlying UNIX implementation
is delivered to the offending process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lm flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-process
memory size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process memory size
limit.
-lM per-request memory space limit [ , warn-limit ]
Set a per-request maximum and an optional
warning cumulative memory space limit for all
processes that constitute the running batch
request. If the sum of all memory consumed by
all of the processes comprising the running
request exceeds the maximum per-request memory
space limit for the request, then all of the
processes in the request will be terminated by a
signal chosen by the underlying UNIX
implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-request warning memory size limits.
When such a warning limit is exceeded, a signal
is delivered to one or more of the processes
comprising the running request, depending upon
the underlying UNIX implementation.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lM flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-request
memory space limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-request memory space
limit.
Set a per-process nice value for all processes
comprising the running batch request.
At present, all UNIX implementations support the
use of an integer called the nice value, which
determines the execution-time priority of a
process relative to all other processes in the
system. By letting the user set a limit on the
nice value for all processes comprising the
running request, a user can cause a request to
consume less (or more) of the CPU resource
presented by the execution machine.
This is particularly useful when a user wishes
to execute a CPU intensive batch request on a
machine running interactive processes. By
setting a low execution-time priority, a user
can make a long running batch request give way
to more interactive processes during the
daytime, while the coming of the nighttime hours
with typically smaller process loads will allow
such a request to gain more and more of the CPU
resource. In this way, long running batch
requests can be polite to their more transient,
interactive neighbor processes.
The only quirk associated with this flag results
from the peculiar choice of nice values,
implemented by the standard UNIX
implementations. In general, increasingly
negative nice values cause the relative
execution priority of a process to increase,
while increasingly positive nice values causes
the relative priority to decrease! Thus, a nice
value limit specification of: "-ln -10" is
greedier than a nice value limit specification
of: "-ln 0".
Since varying UNIX implementations often support
a different finite range of nice values, NQS
allows the specification of nice values that can
eventually turn out to be outside the limits for
the UNIX implementation running at the execution
machine. In such cases, NQS will simply bind
the specified nice value limit to within the
necessary range as appropriate.
Lastly, any nice value specified by the use of
this flag must be acceptable to the batch queue
in which the request is ultimately placed (see
the section entitled LIMITS for more
-ls per-process stack-segment size limit [ , warn-limit ]
Set a per-process maximum and an optional
warning stack-segment size limit for all
processes that constitute the running batch
request. If any process comprising the running
request exceeds the maximum per-process stack-
segment size limit for the request, then that
process is terminated by a signal chosen by the
underlying UNIX implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-process warning stack-segment size
limits. When a warning limit is exceeded, a
signal as determined by the underlying UNIX
implementation is delivered to the offending
process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -ls flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-process
stack-segment size limits. If a batch request
specifies this limit, and the machine upon which
the batch request is eventually run does not
support the enforcement of this limit, then the
limit is simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process stack-segment
size limit.
-lt per-process CPU time limit [ , warn-limit ]
Set a per-process maximum and an optional
warning CPU time limit for all processes that
constitute the running batch request. If any
process comprising the running request exceeds
the maximum per-process CPU time limit for the
request, then that process is terminated by a
implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-process CPU warning time limits.
When a warning limit is exceeded, a signal as
determined by the underlying UNIX implementation
is delivered to the offending process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lt flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-process
CPU time limits. If a batch request specifies
this limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process CPU time limit.
-lT per-request CPU time limit [ , warn-limit ]
Set a per-request maximum and an optional
warning cumulative CPU time limit for all of the
processes that constitute the running batch
request. If the sum of the CPU times consumed
by all of the processes in the batch request
exceeds the maximum per-request CPU time limit
for the request, then all of the processes in
the request will be terminated by a signal
chosen by the underlying UNIX implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-request CPU warning time limits.
When such a warning limit is exceeded, a signal
is delivered to one or more of the processes
comprising the running request, depending upon
the underlying UNIX implementation.
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lT flag with its associated
limit value(s) appears within the batch request
script file.
Not all UNIX implementations support per-request
CPU time limits. If a batch request specifies
this limit, and the machine upon which the batch
request is eventually run does not support the
enforcement of this limit, then the limit is
simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-request CPU time limit.
-lv per-process temporary file size limit [ , warn-limit ]
Set a per-process maximum and an optional
warning temporary (volatile) file size limit for
all processes that constitute the running batch
request. If any process comprising the running
request attempts to write to a temporary file
such that the file size would increase beyond
the maximum per-process temporary-file size
limit for the request, then that process is
terminated by a signal chosen by the underlying
UNIX implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-process warning temporary-file size
limits. When a warning limit is exceeded, a
signal as determined by the underlying UNIX
implementation is delivered to the offending
process.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
limit value(s) appears within the batch request
script file.
At the time of this writing, no UNIX operating
system known to the author supported a
distinction at the kernel level between
permanent and temporary files. Certainly, a
pseudo-temporary file can be constructed by
creating it, and then unlinking its pathname.
However, the file space allocated for such a
file will be allocated from the same pool of
disk space that all other UNIX files are
allocated from.
Until a mechanism is implemented in the kernel
that knows about permanent and temporary files,
this limit cannot be supported in the sense most
useful for batch requests, namely the strict
enforcement of disk quotas for permanent versus
temporary files.
Until such a time, this limit will simply be
ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process temporary-file
size limit.
-lV per-request temporary file space limit [ , warn-limit ]
Set a per-request maximum and an optional
warning cumulative temporary (volatile) file
space limit for all processes that constitute
the running batch request. If any process
comprising the running request attempts to write
to a temporary file such that the file space
consumed by all temporary files opened for
writing by all of the processes in the batch
request would increase beyond the maximum per-
request temporary-file space limit for the
request, then all of the processes in the
request will be terminated by a signal chosen by
the underlying UNIX implementation.
The ability to specify an optional warning limit
exists for those UNIX operating systems that
support per-request warning temporary-file space
limits. When such a warning limit is exceeded,
a signal is delivered to one or more of the
processes comprising the running request,
implementation.
If a maximum limit (and optional warning limit)
specification is comprised of two or more tokens
separated by whitespace characters, then the
specification must be enclosed within double
quotes, or otherwise escaped such that Qsub and
the shell will interpret the entire
specification as a single character string
token. This caveat also applies when an
embedded default -lV flag with its associated
limit value(s) appears within the batch request
script file.
At the time of this writing, no UNIX operating
system known to the author supported a
distinction at the kernel level between
permanent and temporary files. Certainly, a
pseudo-temporary file can be constructed by
creating it, and then unlinking its pathname.
However, the file space allocated for such a
file will be allocated from the same pool of
disk space that all other UNIX files are
allocated from.
Until a mechanism is implemented in the kernel
that knows about permanent and temporary files,
this limit cannot be supported in the sense most
useful for batch requests, namely the strict
enforcement of disk quotas for permanent versus
temporary files.
Until such a time, this limit will simply be
ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a temporary-file space limit.
-lw per-process working set size limit
Set a per-process maximum working set size limit
for all processes that constitute the running
batch request.
Not all UNIX implementations support per-process
working set size limits, and such a limit only
makes sense in the context of a paged virtual
memory machine. If a batch request specifies
this limit, and the machine upon which the batch
request is eventually run does not support the
simply ignored.
See the section entitled LIMITS for more
information on the implementation of batch
request limits, and for a description of the
precise syntax of a per-process working set size
limit.
-mb Send mail to the user on the originating machine
when the request begins execution. If the -mu
flag is also present, then mail is sent to the
user specified for the -mu flag instead of to
the invoking user.
-me Send mail to the user on the originating machine
when the request has ended execution. If the
-mu flag is also present, then mail is sent to
the user specified for the -mu flag instead of
to the invoking user.
-mr Send mail to the user on the originating machine
when the request is restarted. If the -mu flag
is also present, then mail is sent to the user
specified for the -mu flag instead of to the
invoking user.
-mt Send mail to the user on the originating machine
when the request is transferred to the execution
machine. If the -mu flag is also present, then
mail is sent to the user specified for the -mu
flag instead of to the invoking user.
-mu user-name
Specify that any mail concerning the request
should be delivered to the user user-name.
User-name may be formatted either as user
(containing no `@' characters), or as
user@machine. In the absence of this flag, any
mail concerning the request will be sent to the
invoker on the originating machine.
-nr Declare that the request is non-restartable. If
this flag is specified, then the request will
not be restarted by NQS upon system boot if the
request was running at the time of an NQS
shutdown or system crash.
Requests that are not running are always
preserved across host crashes and NQS shutdowns
for later requeueing, with or without this flag.
the qmgr(1M) NQS control program, a SIGTERM
signal is sent to all processes comprising all
running NQS requests on the local host, and all
queued NQS requests are barred from beginning
execution. After a finite number of seconds
have elapsed (typically sixty, but this value
can be overridden by the operator), all
remaining processes comprising all remaining
running NQS requests are killed by the signal:
SIGKILL.
For an NQS request to be properly restarted
after an NQS shutdown, the -nr flag must not be
specified, and the spawned batch request shell
must ignore SIGTERM signals (which is done by
default). The spawned batch request shell must
also not exit before the final SIGKILL arrives.
Since the batch request shell is simply spawning
commands and programs, waiting for their
completion, this implies that the commands and
programs being executed by the batch request
shell must also be immune to SIGTERM signals,
saving state as appropriate before being killed
by the final SIGKILL signal.
See the CAVEATS section below for more
discussion concerning the restartability of
batch requests.
-o [machine:][[/]path/] stdout-filename
Direct output generated by the batch request
which is sent to the stdout file to the named
[machine:][[/]path/] stdout-filename.
The brackets "[" and "]" enclose optional
portions of the stdout destination machine,
path, and stdout-filename.
If no explicit machine destination is specified,
then the destination machine defaults to the
machine that originated the batch request, or to
the machine that will eventually run the
request, depending on the respective absence, or
presence of the -ko flag.
If no machine destination is specified, and the
path/filename does not begin with a "/", then
the current working directory is prepended to
create a fully qualified path name, provided
that the -ko (keep stdout) flag is also absent.
In all other cases, any partial path/filename is
directory on the stdout destination machine.
If no -o [machine:][[/]path/] stdout-filename
flag is specified, then all stdout output for
the batch request is sent to the file whose name
consists of the first seven characters of the
request-name followed by the characters: ".o",
followed by the request sequence number portion
of the request-id discussed below. In the
absence of the -ko flag, this default stdout
output file will be placed on the machine that
originated the batch request in the current
working directory, as defined when the batch
request was first submitted. Otherwise, the
file will be placed in the user's home directory
on the execution machine.
If the stdout-filename contains any #
characters, they will be replaced by the numeric
part of the request's sequence number. This
provides extra flexibility in naming the stdout
output file. The # character is ignored in all
other parts of the file path.
-p priority Explicitly assign an intra-queue priority to the
request. The specified priority must be an
integer, and must be in the range [0..63],
inclusive. A value of 63 defines the highest
intra-queue request priority, while a value of 0
defines the lowest. This priority does not
determine the execution priority of the request.
This priority is only used to determine the
relative ordering of requests within a queue.
When a request is added to a queue, it is placed
at a specific position within the queue such
that it appears ahead of all existing requests
whose priority is less than the priority of the
new request. Similarly, all requests with a
higher priority will remain ahead of the new
request when the queueing process is complete.
When the priority of the new request is equal to
the priority of an existing request, the
existing request takes precedence over the new
request.
If no intra-queue priority is chosen by the
user, then NQS assigns a default value.
-q queue-name
Specify the queue to which the batch request is
specification is given, then the user's
environment variable set is searched for the
variable: QSUB_QUEUE. If this environment
variable is found, then the character string
value for QSUB_QUEUE is presumed to name the
queue to which the request should be submitted.
If the QSUB_QUEUE environment variable is not
found, then the request will be submitted to the
default batch request queue, if defined by the
local system administrator. Otherwise, the
request cannot be queued, and an appropriate
error message is displayed to this effect.
-r request-name
Assign the specified request-name to the
request. In the absence of an explict -r
request-name specification, the request-name
defaults to the name of the script file (leading
path name removed) given on the command line.
If no script file was given, then the default
request-name assigned to the request is STDIN.
In all cases, if the request-name is found to
begin with a digit, then the character 'R' is
prepended to prevent a request-name from
beginning with a digit. All request-names are
truncated to a maximum length of 15 characters.
-re By default, all output generated by a batch
request sent to the stderr file is temporarily
into a file residing in a protected directory on
the machine that executes the request. When the
batch request completes execution, this file is
then spooled to its final destination, possibly
on a remote machine.
This default spooling of the stderr output file
is done to reduce the network traffic costs
incurred by the submitter (owner) of a batch
request which is to return its stderr output to
a remote machine upon completion. In some
cases, this behavior is not desired. If it is
necessary to override this behavior, then the
-re flag can be specified which says that stderr
output produced by the request is to be
immediately written to the final destination
file, as output is generated, no matter what the
networking cost.
Circumstances may not allow a given NQS
implementation to support this flag, in which
file will simply be spooled as it ordinarily
would without this flag.
-ro By default, all output generated by a batch
request sent to the stdout file is temporarily
spooled into a file residing in a protected
directory on the machine that executes the
request. When the batch request completes
execution, this file is then spooled to its
final destination, possibly on a remote machine.
This default spooling of the stdout output file
is done to reduce the network traffic costs
incurred by the submitter (owner) of a batch
request which is to return its stdout output to
a remote machine upon completion. In some
cases, this behavior is not desired. If it is
necessary to override this behavior, then the
-ro flag can be specified which says that stdout
output produced by the request is to be
immediately written to the final destination
file, as output is generated, no matter what the
networking cost.
Circumstances may not allow a given NQS
implementation to support this flag, in which
case it will be ignored, and the stdout output
file will simply be spooled as it ordinarily
would without this flag.
-rs Indicate that the request is restartable. By
default requests are not restartable. If this
switch is used, it is the responsibility of the
user to ensure that the request will execute
correctly if restarted, by the use of
appropriate programming techniques.
-s shell-name
Specify the absolute path name of the shell
which will be used to interpret the batch
request script. This flag unconditionally
overrides any shell strategy configured on the
execution machine for selecting which shell to
spawn in order to interpret the batch request
script.
In the absence of this flag, the NQS system at
the execution machine will use one of three (3)
distinct shell choice strategies for the
execution of the batch request. Any one of the
three strategies can be configured by a system
The three shell strategies are called:
fixed,
free, and
login.
These shell strategies respectively cause the
configured fixed shell to be exec'd to interpret
all batch requests, cause the user's login shell
as defined in the password file to be exec'd
which in turn chooses and spawns the appropriate
shell for interpreting the batch request script,
or cause only the user's login shell to be
exec'd to interpret the script.
A shell strategy of fixed means that the same
shell (as configured by the system
administrator), will be used to execute all
batch requests.
A shell strategy of free will run the batch
request script exactly as would an interactive
invocation of the script, and is the default NQS
shell strategy.
The strategies of fixed and login exist for host
systems that are short on available free
processes. In these two strategies, a single
shell is exec'd, and that same shell is the
shell that executes all of the commands in the
batch request script.
The shell strategy configured for a particular
NQS system can be determined by the qlimit(1)
command.
-v Print the current version of the Qsub program.
-x Export all environment variables. When a batch
request is submitted, the current values of the
environment variables: HOME, SHELL, PATH,
LOGNAME (not all systems), USER (not all
systems), MAIL, and TZ are saved for later
recreation when the batch request is spawned, as
the respective environment variables: QSUB_HOME,
QSUB_SHELL, QSUB_PATH, QSUB_LOGNAME, QSUB_USER,
QSUB_MAIL, and QSUB_TZ. Unless the -x flag is
specified, no other environment variables will
be exported from the originating host for the
batch request. If the -x flag option is
variables whose names do not conflict with the
automatically exported variables, are also
exported with the request. These additional
environment variables will be recreated under
the same name when the batch request is spawned.
-z Submit the batch request silently. If the
request is submitted successfully, then no
messages are displayed indicating this fact.
Error messages will, however, always be
displayed.
If the batch request is successfully submitted and the -z
flag has not been specified, the request-id of the request
is displayed to the user. A request-id is always of the
form: seqno.hostname where seqno refers to the sequence
number assigned to the request by NQS, and hostname refers
to the name of originating local machine. This identifier
is used throughout NQS to uniquely identify the request, no
matter where it is in the network.
The following events take place in the following order when
an NQS batch request is spawned:
The process that will become the head of the
process group for all processes comprising the
batch request is created by NQS.
Resource limits are enforced.
The real and effective group-id of the process
is set to the group-id as defined in the local
password file for the request owner.
The real and effective user-id of the process is
set to the real user-id of the batch request
owner.
The user file creation mask is set to the value
that the user had on the originating machine
when the batch request was first submitted.
It the user explicitly specified a shell by use
of the -s flag (discussed above), then that
user-specified shell is chosen as the shell that
will be used to execute the batch request
script. Otherwise, a shell is chosen based upon
the shell strategy as configured for the local
NQS system (see the earlier discussion of the -s
flag for a description of the possible shell
strategies that can be configured for an NQS
The environment variables of HOME, SHELL, PATH,
LOGNAME (not all systems), USER (not all
systems), and MAIL are set from the user's
password file entry, as though the user had
logged directly into the execution machine.
The environment string: ENVIRONMENT=BATCH is
added to the environment so that shell scripts
(and the user's .profile (Bourne shell) or
.cshrc and .login (C-shell) scripts), can test
for batch request execution when appropriate,
and not (for example) perform any setting of
terminal characteristics, since a batch request
is not connected to an input terminal.
The environment variables of QSUB_WORKDIR,
QSUB_HOST, QSUB_REQNAME, and QSUB_REQID are
added to the environment. These environment
variables equate to the obvious respective
strings of the working directory at the time
that the request was submitted, the name of the
originating host, the name of the request, and
the request request-id.
All of the remaining environment variables saved
for recreation when the batch request is spawned
are added at this point to the environment.
When a batch request is initially submitted, the
current values of the environment variables:
HOME, SHELL, PATH, LOGNAME (not all systems),
USER (not all systems), MAIL, and TZ are saved
for later recreation when the batch request is
spawned. When recreated however, these
variables are added to the environment under the
respective names: QSUB_HOME, QSUB_SHELL,
QSUB_PATH, QSUB_LOGNAME, QSUB_USER, QSUB_MAIL,
and QSUB_TZ, to avoid the obvious conflict with
the local version of these environment
variables. Additionally, all environment
variables exported from the originating host by
the -x option are added to the environment at
this time.
The current working directory is then set to the
user's home directory on the execution machine,
and the chosen shell is exec'd to execute the
batch request script with the environment as
constructed in the steps outlined above.
In all cases, the chosen shell is exec'd as though it were
the script, then the .profile file is read. If the C-shell
is chosen, then the .cshrc and .login scripts are read.
If the user did not specify a specific shell for the batch
request, then NQS chooses which shell should be used to
execute the shell script, based on the shell strategy as
configured by the system administrator (see the earlier
discussion of the -s flag).
In such a case, a free shell strategy instructs NQS to
execute the login shell for the user (as configured in the
password file). The login shell is in turn instructed to
examine the shell script file, and fork another shell of the
appropriate type to interpret the shell script, behaving
exactly as an interactive invocation of the script.
Otherwise no additional shell is spawned, and the chosen
fixed or login shell sequentially executes the commands
contained in the shell script file until completion of the
batch request.
QUEUE TYPES
NQS supports four different queue types that serve to
provide four very different functions. These four queue
types are known as batch, device, pipe, and network.
The queue type of batch can only be used to execute NQS
batch requests. Only NQS batch requests created by the
qsub(1) command can be placed in a batch queue.
The queue type of device can only be used to execute NQS
device requests. Only NQS device requests created by the
qpr(1) command can be placed in a device queue.
Queues of type pipe are used to send NQS requests to other
pipe queues, or to request destination queues of type batch
or device, as appropriate for the request type. In general,
pipe queues, in combination with network queues, act as the
mechanism that NQS uses to transport both batch and device
requests to distant queues on other remote machines. It is
also perfectly legal for a pipe queue to transport requests
to queues on the same machine.
When a pipe queue is defined, it is given a destination set
which defines the set of possible destination queues for
requests entered in that pipe queue. In this manner, it is
possible for a batch or device request to pass through many
pipe queues on its way to its ultimate destination, which
must eventually be a queue of type batch or device (matching
the request type).
handled by a pipe queue, the associated server is spawned
which must select a queue destination for the request being
handled, based on the characteristics of the request, and
upon the characteristics of each queue in the destination
set defined for the pipe queue.
Since a different server can be configured for each pipe
queue, and batch and device queues can be endowed with the
pipeonly attribute that will only admit requests queued via
another pipe queue, it is possible for respective NQS
installations to use pipe queues as a request class
mechanism, placing requests that ask for different resource
allocations in different queues, each of which can have
different associated limits and priorities.
It is also completely possible for a pipe queue server, when
handling a request, to discover that no destination queue
will accept the request, for various reasons which can
include insufficient resource limits to execute the request,
or a lack of a corresponding account or privilege for
queueing at a remote queue. In such circumstances, the
request will be deleted, and the user will be notified by
mail (see mail(1)).
The queue type of network, as alluded to earlier, is
implicitly used by pipe queues to pass NQS requests between
machines, and is also used to handle queued file transfer
operations.
QUEUE ACCESS
NQS supports queue access restrictions. For each queue of
queue type other than network, access may be either
unrestricted or restricted. If access is unrestricted, any
request may enter the queue. If access is restricted, a
request can only enter the queue if the requester or the
requester's login group has been given access to that queue
(see qmgr(1M)). Requests submitted by root are an
exception; they are always queued, even if root has not
explicitly been given access.
Use qstat(1) to determine who has access to a particular
queue.
LIMITS
NQS supports many batch request resource limit types that
can be applied to an NQS batch request. The existence of
configurable resource limits allows an NQS user to set
resource limits within which his or her request must
execute. In many instances, smaller limit values can result
in a more favorable scheduling policy for a batch request.
limit-flags (-llimit-letter-type), is quite flexible, and
describes values for two general limit categories. These
two general categories respectively deal with time related
limits, and those limits are not time related.
For finite CPU time limits, the limit-value is expressed in
the reasonably obvious format:
[[hours :] minutes : ] seconds [.milliseconds]
Whitespace can appear anywhere between the principal tokens,
with the exception that no whitespace can appear around the
decimal point.
Example time limit-values are:
1234 : 58 : 21.29 - 1234 hrs 58 mins 21.290 secs
12345 - 12345 seconds
121.1 - 121.100 seconds
59:01 - 59 minutes and 1 second
For all other finite limits (with the exclusion of the nice
limit-value -ln), the acceptable syntax is:
.fraction [units]
or
integer [.fraction] [units]
where the integer and fraction tokens represent strings of
up to eight decimal digits, denoting the obvious values. In
both cases, the units of allocation may also be specified as
one of the case insensitive strings:
b - bytes
w - words
kb - kilobytes (2^10 bytes)
kw - kilowords (2^10 words)
mb - megabytes (2^20 bytes)
mw - megawords (2^20 words)
gb - gigabytes (2^30 bytes)
gw - gigawords (2^30 words)
In the absence of any units specification, the units of
bytes are assumed.
For all limit types with the exception of the nice limit-
value (-ln), it is possible to state that no limit should be
applied. This is done by specifying a limit-value of
"unlimited", or any initial substring thereof. Whenever an
type, then the batch request operates as though no explicit
limits have been placed upon the corresponding resource,
other than by the limitations of the physical hardware
involved.
The complications caused by batch request resource limits
first show up when queueing a batch request in a batch
queue. This operation is described in the following
paragraphs.
If a batch request specifies a limit that cannot be enforced
by the underlying UNIX implementation, then the limit is
simply ignored, and the batch request will operate as though
there were no limit (other than the obvious physical
maximums), placed upon that resource type. (See the
qlimit(1) command to find out what limits are supported by a
given machine.)
For each remaining finite limit that can be supported by the
underlying UNIX implementation that is not a CPU time-limit
or UNIX execution-time nice-value-limit, the limit-value is
internally converted to the units of bytes or words,
whichever is more appropriate for the underlying machine
architecture.
As an example, a per-process memory size limit value of 321
megabytes would be interpreted as 321 x 2^20 bytes, provided
that the underlying machine architecture was capable of
directly addressing single bytes. Thus the original limit
coefficient of 321 would become 321 x 2^20. On a machine
that was only capable of addressing words, the appropriate
conversion of 321 x 2^20 bytes / #of-bytes-per-word would be
performed.
If the result of such a conversion would cause overflow when
the coefficient was represented as a signed-long integer on
the supporting hardware, then the coefficient is replaced
with the coefficient of: of 2^N-1 where N is equal to the
number of bits of precision in a signed long integer. For
typical 32-bit machines, this default extreme limit would
therefore be 2^31-1 bytes. For word addressable machines in
the supercomputer class supporting 64-bit long integers, the
default extreme limit would be 2^63-1 words.
Lastly, some implementations of UNIX reserve coefficients of
the form: 2^N-1 as synonymous with infinity, meaning no
limit is to be applied. For such UNIX implementations, NQS
further decrements the default extreme limit so as not to
imply infinity.
The identical internal conversion process as described in
limit-value configured for a particular batch queue using
the qmgr(1M) program.
After all of the applicable limit-values have been converted
as described above, each such resulting limit-value is then
compared against the corresponding limit-value as configured
for the destination batch queue. If, for every type of
limit, the batch queue limit-value is greater than or equal
to the corresponding batch request limit-value, then the
request can be successfully queued, provided that no other
anomalous conditions occur. For request infinity limit-
values, the corresponding queue limit-value must also be
configured as infinity.
These resource limit checks are performed irrespective of
the batch request arrival mechanism, either by a direct use
of the qsub(1) command, or by the indirect placement of a
batch request into a batch queue via a pipe queue. It is
impossible for a batch request to be queued in an NQS batch
queue if any of these resource limit checks fail.
Finally, if a request fails to specify a limit-value for a
resource limit type that is supported on the execution
machine, then the corresponding limit-value configured for
the destination queue becomes the limit-value for the
unspecified request limit.
Upon the successful queueing of a request in a batch queue,
the set of limits under which the request will execute is
frozen, and will not be modified by subsequent qmgr(1M)
commands that alter the limits of the containing batch
queue.
CAVEATS
When an NQS batch request is spawned, a new process-group is
established such that all processes of the request exist in
the same process-group. If the qdel(1) command is used to
send a signal to an NQS batch request, the signal is sent to
all processes of the request in the created process-group.
However, should one or more processes of the request choose
to successfully execute a setpgrp(2) system call, then such
processes will not receive any signals sent by the qdel(1)
command. This can lead to "rogue" requests whose
constituent processes must be killed by other means such as
the kill(1) command. However, NQS takes advantage of any
UNIX implementations that provide a mechanism of "locking" a
process, and all of its subsequent children in a particular
process-group. For such UNIX implementations, this problem
does not occur.
It is extremely wise for all processes of an NQS request to
SIGTERM signal causes the receiving process to die. NQS
sends a SIGTERM signal to all processes in the established
process-group for a batch request as a notification that the
request should be prepared to be killed, either because of
an abort queue command issued by an operator using the
qmgr(1M) program, or because it is necessary to shutdown NQS
and all running requests as part of a general shutdown
procedure of the local host.
It must be understood that the spawned shell ignores SIGTERM
signals. If the current immediate child of the shell does
not ignore or catch SIGTERM signals, then it will be killed
by the receipt of such, and the shell will go on to execute
the next command from the script (if there is one). In any
case, the shell will not be killed by the SIGTERM signal,
though the executing command will have been killed.
After receiving a SIGTERM signal delivered from NQS, a
process of a batch request typically has sixty seconds to
get its "house in order" before receiving a SIGKILL signal
(though the sixty second duration can be changed by the
operator).
All batch requests terminated because of an operator NQS
shutdown request that did not specify the -nr flag are
considered restartable by NQS, and are requeued (provided
that the batch request shell process is still present at the
time of the SIGKILL signal broadcast as discussed above), so
that when NQS is rebooted, such batch requests will be
respawned to continue execution. It is however, up to the
user to make the request restartable by the appropriate
programming techniques. NQS simply spawns the request again
as though it were being spawned for the first time.
Upon completion of a batch request, a mail message can be
sent to the submitter (see the discussion of the -me flag
above). In many instances, the completion code of the
spawned Bourne or C-Shell will be displayed. This is merely
the value returned by the shell through the exit(2) system
call.
Lastly, there is no good way to echo commands executed by
unmodified versions of the Bourne and C shells. While the
C-shell can be spawned in such a fashion as to echo the
commands it executes, it is often very difficult to tell an
echoed command from genuine output produced by the batch
request, because no "magic" character such as a '$' is
displayed in front of the echoed command. The Bourne shell
does not support any echo option whatsoever.
Thus, one of the better ways to write the shell script for a
script of the form:
echo "explanatory-message"
where the echoed message should be a meaningful message
chosen by the user.
LIMITATIONS AND IMPLEMENTATION NOTES
Network queues have not yet been implemented.
The original "@$" syntax used to introduce embedded argument
flags was chosen because it rarely conflicts with anything
else present in a shell script file.
SEE