CLI Tools

Several CLI tools are provided with OCS. This page describes these tools, what they are used for, and how to use them.

ocsbow

The purpose of ocsbow is to issue commands to HostManager Agents. See Centralized Management for general instructions.

ocsbow should be able to function in any environment where you can instantiate an OCSClient; it needs to know how to find the site config file, and it needs access to the crossbar router.

Command-line arguments

(The output from ocsbow --help should be rendered here.)

ocsbow is used to talk to HostManager agents across an OCS installation. In a distributed OCS, you can request that Agents across the observatory be started or stopped.

usage: ocsbow [-h] {status,up,down,config} ...

subcommands

For more information and options pertinent to a subcommand, add –help (e.g., “ocsbow up –help”).

command: Possible choices: status, up, down, config

Sub-commands

status

Status information is obtained by querying all HostManagers described in the local version of the site config file. The output is tabulated by host, and lists each agent reported as being managed by the HostManager (which may include agents not listed in the local copy of the site config).

ocsbow status [-h] [--host HOST] [--reload-config]

Named Arguments

--host, -H: Limit hosts that are displayed.
--reload-config, -r: Request each HM to reprocess the site config file.

up

Mark targets as ‘up’ (so HostManagers will launch them).

ocsbow up [-h] [--all] [--dry-run] [instance ...]

Positional Arguments

instance: instance-id to target. If this is the id of a HostManager agent, it will be asked to start all of its managed agents.

Named Arguments

--all, -a

Apply the command to all HostManagers in the OCS.

Default: False

--dry-run

If set, HostManagers will be queried for info but no state change requests will be issued.

Default: False

down

Mark targets as ‘down’ (so HostManagers will shut them down).

ocsbow down [-h] [--all] [--dry-run] [instance ...]

Positional Arguments

instance: instance-id to target. If this is the id of a HostManager agent, it will be asked to start all of its managed agents.

Named Arguments

--all, -a

Apply the command to all HostManagers in the OCS.

Default: False

--dry-run

If set, HostManagers will be queried for info but no state change requests will be issued.

Default: False

config

Query the local OCS config.

ocsbow config [-h] [{summary,plugins,crossbar}]

Positional Arguments

cfg_request

Possible choices: summary, plugins, crossbar

Default: “summary”

ocs-local-support

This script helps with launching crossbar and/or a local HostManager instance. It is useful in small (e.g. single host systems) or as an alternative to managing crossbar at the system level or with Docker.

(The output from ocs-local-support --help should be rendered here.)

ocs-local-support is used to control a crossbar router and HostManager agent on this host.

Examples:

ocs-local-support status
ocs-local-support start
ocs-local-support stop

Or with a target subsystem:

ocs-local-support stop crossbar
ocs-local-support status agent
ocs-local-support start process

usage: ocs-local-support [-h] [--foreground]
                         {status,start,stop,generate_crossbar_config}
                         [{crossbar,agent,process}]

Positional Arguments

here_cmd

Possible choices: status, start, stop, generate_crossbar_config

Command to apply to the targets.

target

Possible choices: crossbar, agent, process

Operate on the specific subsystem only.

Named Arguments

--foreground

For targeted ‘start’, run the command in the foreground and copy stdout/stderr to the terminal.

Default: False

checkdata

checkdata is a CLI tool for quickly viewing the most recent times each agent instance-id’s feeds and associated fields were seen in the data written to disk. This is meant as a quick way to check on the state of data aggregation.

Note

checkdata will open all files within a directory, walking as deep as it needs to to find all .g3 files within. It’s recommended not to run it on an entire data output directory, unless you want to wait for a long time. Instead, you should probably just run it on a single day’s directory, or even individual file, depending on your needs.

Note

If so3g is not installed on your system, checkdata will use a docker image. This will require permissions to interact with docker.

For info on how to run, see the help:

$ checkdata -h
usage: checkdata [-h] [--verbose] [--docker] target

positional arguments:
  target         File or directory to scan.

optional arguments:
  -h, --help     show this help message and exit
  --verbose, -v
  --docker, -d   Force use of docker, even if so3g is installed.

Usage/Examples

To use checkdata simply invoke it on the commandline:

$ checkdata /data/15732/
Scanning |################################| 11/11 Processing |################################| 156/156

LSA22YE
  temperatures: 643.6 s old

LSA22YG
  temperatures: 727.9 s old

LSA22Z2
  temperatures: 310.0 s old

LSA22ZC
  temperatures: 318.9 s old

LSA24R5
  temperatures: 286.6 s old

LSA2761
  temperatures: 355.4 s old

bluefors
  bluefors: 371.3 s old

Note

This assumes checkdata is in your user’s $PATH.

This presents each agent, the agent’s feeds, and a time representing the oldest field within that feed. For more verbose output, throw the -v flag:

$ checkdata -v /data/15732/2019-11-08-10-35-29.g3
Scanning |################################| 1/1
Processing |################################| 156/156

LSA22YG
  temperatures
  -----------------------------------------------------------------------------------------
                 Field |    Last Seen [s ago] |      Seen At [ctime] |                Value
  -----------------------------------------------------------------------------------------
          Channel 01 R |                320.4 |   1573212564.6585813 |              29255.1
          Channel 01 T |                320.4 |   1573212564.6585813 |            0.0656435

Normal output from checkdata will show in your default terminal color scheme. When fields were last seen more than 10 minutes ago their age will show up in red. If a field name is invalid, it will show up in yellow in the verbose output.

datestring2ctime

The HK Aggregator originally output .g3 files with the naming convention %Y-%m-%d-%H-%M-%S.g3. After some time we decided to move to a ctime based filename, i.e. 1582661596.g3. To facilitate the move the datestring2ctime script was created. It will rename all datestring based .g3 files in a given directory to the ctime convention.

Warning

The script is safe to run, but be aware of what you are doing, and that is renaming every .g3 file matching the old convention. This has the potential to break scripts you have written that read in files, especially if that do any parsing of the names.

To use the script run:

./datestring2ctime target -v

The passed target can be a single file or directory. The -v flag indicates you’d like verbose output, however this is not required. Without it there will be no output.

g32influx

g32influx is a script which uploads data from .g3 files on disk to InfluxDB. This may be used to restore a database from .g3 file, or upload individual files for browsing.

For information on how to run:

$ ./g32influx -h
usage: g32influx [-h] [--start START] [--end END] [--log LOG] [--logfile LOGFILE] target database host port

positional arguments:
  target                File or directory to scan.
  database              InfluxDB database to publish data to.
  host                  InfluxDB host.
  port                  InfluxDB port.

optional arguments:
  -h, --help            show this help message and exit
  --start START         Set startdate, cutting all files that start before this date.
  --end END             Set enddate, cutting all files that start after this date.
  --log LOG, -l LOG     Set loglevel.
  --logfile LOGFILE, -f LOGFILE
                        Set the logfile.

Note

An SQLiteDB file is used to track which files were uploaded to InfluxDB. This is meant to only avoid reuploading already pushed data, particularly valuable if you need to restart a large upload job. This will be .g32influx.db in the directory you run the script from.

ocs-client-cli

Note

The output from ocs-client-cli --help should be rendered here. In addition to the options discussed, the script supports the same “Site Config Options” that Agents usually support, such as --site-file=.... If there are some stray instances of %(prog)s, imagine ocs-client-cli in their place.)

Note

To learn about using an OCS Client and writing control programs, please see Clients and Control Programs.

This script provides a quick way to start a python or ipython shell to interact with an OCS Agent. You will need to set OCS_CONFIG_DIR environment variable to the directory containing default.yaml, or else use –site-* options to specify your configuration.

If you know the instance-id of the Agent you want to talk to, run:

%(prog)s shell INSTANCE_ID

If you want to listen to heartbeat feeds to get a list of Agents in the system, run:

%(prog)s scan

usage: ocs-client-cli [-h] command ...

Positional Arguments

command

Possible choices: shell, scan, listen

Command (“ocs-client-cli {command} -h” for more help)

Sub-commands

shell

Start an interactive python session with an OCSClient instantiated.

ocs-client-cli shell [-h] [--simple] [instance_id ...]

Positional Arguments

instance_id: E.g. aggregator or fakedata-1. Pass more than one to get multiple clients.

Named Arguments

--simple

Do not use ipython for the shell, even if it is available.

Default: False

scan

Gather and print list of Agents.

ocs-client-cli scan [-h] [--details] [--use-registry [USE_REGISTRY]]

Named Arguments

--details

List all Operations with their current status OpCode.

Default: False

--use-registry

Query the registry (faster than listening for heartbeats). Pass the registry instance_id as an argument (default to ‘registry’).

listen

Subscribe to feed(s) and dump to stdout.

ocs-client-cli listen [-h] feed_selector

Positional Arguments

feed_selector: Feed name, which can include wildcard matching (double ..). E.g., try ‘observatory..feeds.heartbeat’

ocs-install-systemd

This script assists with setting up systemd services to launch HostManager Agent instances. See Centralized Management for general instructions.

Note

The output from ocs-install-systemd --help should be rendered here. In addition to the options discussed, the script supports the same “Site Config Options” that Agents usually support, such as --site-file=....

usage: ocs-install-systemd [-h] [--docker-compose DOCKER_COMPOSE]
                           [--shell SHELL] [--python-bin PYTHON_BIN]
                           [--launcher-dir LAUNCHER_DIR]
                           [--launcher-script LAUNCHER_SCRIPT]
                           [--service-dir SERVICE_DIR]
                           [--service-user SERVICE_USER]
                           [--service-name SERVICE_NAME]
                           [--service-host SERVICE_HOST]

Options specific to systemd service creation

--docker-compose: Default: []
--shell: Override the shell used for the launcher script.
--python-bin: Select the Python interpreter with which to launch HostManager.
--launcher-dir: Override the directory of the launcher script.
--launcher-script: Override the name of the launcher script.
--service-dir: Directory to which the .service file will be written; default is /etc/systemd/system/.
--service-user: Override the user under which HostManager agent will be run.
--service-name: Override the name of the systemd service.
--service-host: Set a hostname to use when generating the service name.

ocs-agent-cli

This script provides a quick way to start an OCS Agent. You will need to set OCS_CONFIG_DIR environment variable to the directory containing default.yaml, or else use --site-* options to specify your configuration.

To start an Agent, run:

ocs-agent-cli --instance-id INSTANCE_ID

ocs-agent-cli will also inspect environment variables for commonly passed arguments to facilitate configuration within Docker containers. Those environment variables, if defined and non-trivial, will be passed to the agent script unless they are overridden explicitly on the ocs-agent-cli command line. The environment variables and arguments are:

INSTANCE_ID, will be the default value for --instance-id

SITE_HOST, for --site-host

SITE_HUB, for --site-hub

SITE_HTTP, for --site-http

CROSSBAR_TIMEOUT, for --crossbar-timeout

ocs-agent-cli relies on the Agent being run belonging to an OCS Plugin. If the Agent is not an OCS Plugin it can be run directly using both the --agent and --entrypoint flags. For example:

ocs-agent-cli --agent my_agent.py --entrypoint main --instance-id my-agent-1

usage: ocs-agent-cli [-h] [--instance-id INSTANCE_ID] [--site-hub SITE_HUB]
                     [--site-http SITE_HTTP] [--site-host SITE_HOST]
                     [--crossbar-timeout CROSSBAR_TIMEOUT] [--agent AGENT]
                     [--entrypoint ENTRYPOINT]

Named Arguments

--instance-id: Agent unique instance-id. E.g. ‘aggregator’ or ‘fakedata-1’.
--site-hub: Site hub address.
--site-http: Site HTTP address.
--site-host: Declare the host the instance is configured in.
--crossbar-timeout: Length of time in seconds that the Agent will try to reconnect to the crossbar server before shutting down. Disable the timeout by setting to 0.
--agent: Path to non-plugin OCS Agent.
--entrypoint: Agent module entrypoint function.