OCS Site Configuration

Overview

The purpose of ocs.site_config module is to help Agents and Control Clients self-configure within a particular “site” context. By “site”, we mean the set of instances of Agents and Clients, spread across one or more hosts, that should form a single OCS network.

In an observatory or lab, it will be convenient if Agents and Clients can determine the appropriate WAMP server and realm without user intervention. Furthermore, in cases where the same Agent code is launched in multiple instances (to control distinct instances of the same hardware model, for example) there needs to be a system that assigns a WAMP base address (such as detlab.cryo.thermo2) to each Agent instance in a consistent matter.

The approach adopted here is that ocs.site_config works with ArgumentParser to both consume and then modify the arguments being used to launch an Agent instance. Roughly speaking, the following occurs:

When launching an Agent, the user uses command line arguments to specify a site configuration file. If no arguments are set, a sensible default is chosen.
The configuration file is parsed and the special configuration appropriate to this Agent instance is identified. The configuration is determined based on the host the Agent instance is running on, and possibly by an “instance-id”, which helps distinguish between independent Agents running from the same source script.
The special configuration parameters for this Agent instance are merged into the command line parameters.
Through normal processing of command line parameters in the Agent script, the appropriate WAMP connections, address registrations, and hardware associations are made.

ocs.site_config

This section documents in detail the classes in ocs.site_config. For details about the site configuration file, please refer to OCS Site Config File.

SiteConfig

At root level, the configuration file should encode a SiteConfig object. The structure is described in the from_dict method of the SiteConfig class:

class ocs.site_config.SiteConfig[source]

classmethod from_dict(data)[source]

Parameters:: data – The configuration dictionary.

The configuration dictionary should have the following elements:

hub (required): Describes what WAMP server and realm Agents and Clients should use.
hosts (required): A dictionary of HostConfig descriptions. The keys in this dictionary can be real host names on the network, pseudo-host names, or the special value “localhost”.

A HostConfig marked for “localhost” will match any host that does not have an exact match in the hosts dictionary. This should normally be used only in single-host test systems or examples.

Client programs will normally (i.e., by default) try to load the HostConfig associated with the system hostname (that which is returned by socket.gethostname()). But this can be overridden easily, for example by using the --site-host command line argument. It is thus quite reasonable to use the hosts dictionary to hold a set of useful configurations indexed by a user-specified string (a pseudo-host).

The hub information is used by all Agent and Control Clients, on all hosts, to connect to the OCS WAMP router. This WAMP router (probably crossbar) usually has its own configuration file. The settings in SCF hub block are parsed by the from_dict method of the HubConfig class:

class ocs.site_config.HubConfig[source]

classmethod from_dict(data, parent=None)[source]

Parameters:

data – The configuration dictionary.
parent – the SiteConfig from which this data was extracted (this is stored as self.parent, but not used).

The configuration dictionary should have the following elements:

wamp_server (required): URL to the WAMP router’s websocket: access point for ocs. E.g., ws://host-2:8001/ws. WAMP routers can have multiple access points, with different protocols, security layers, and permissions. (Command line override: --site-hub.)
wamp_http (optional): URL to the WAMP router’s http bridge: interface. This is the best interface for simple clients to use. E.g., http://host-2:8001/call.
wamp_realm (required): The WAMP realm to use. WAMP: clients operating in a particular realm are isolated from clients connected to other realms. Example and test code will often use debug_realm here. (Command line override: --site-realm.)
address_root (required): The base address to be used by: all OCS Agents. This is normally something simple like observatory or detlab. (Command line override: --address-root.)
access_policy (optional): Specify the default access: policy for all agents. To tell all agents to contact an Access Director Agent with instance_id access-dir, set this to the string “director:access-dir”. (Command line override: --access-policy.)

HostConfig

The structure of HostConfig encoding is described in the from_dict method of the HostConfig class:

class ocs.site_config.HostConfig(name=None)[source]

classmethod from_dict(data, parent=None, name=None)[source]

Parameters:

data – The configuration dictionary.
parent – the SiteConfig from which this data was extracted (this is stored as self.parent, but not used).

The configuration dictionary should have the following elements:

agent-instances (required): A list of AgentConfig descriptions.
crossbar (optional): Settings to assist with starting / stopping / monitoring a crossbar server running on this host. There is a single crossbar server for an OCS network and thus this entry should be defined for at most one of the hosts in the site config file. Note that setting this to None (or null) will disable host crossbar control, while setting it to an empty dictionary, {}, will enable local host crossbar control with default settings.
log-dir (optional): Path at which to write log files. Relative paths will be interpreted relative to the “working directory”; see –working-dir command line option.

To allow ocs to manipulate the crossbar router (e.g. if you want to easily start and stop it using ocsbow), then the crossbar variable should be defined, with (at least) an empty dictionary. The details of the options are described in the from_dict method of the CrossbarConfig class:

class ocs.site_config.CrossbarConfig[source]

classmethod from_dict(data, parent=None)[source]

Parameters:

data – The configuration dictionary, or None.
parent – the HostConfig from which this data was extracted (this is stored as self.parent, but not used).

The configuration dictionary should have the following elements:

config-dir (optional): Location of crossbar config.json;: this gets passed to --cbdir, if specified..
bin (optional): The path to the crossbar executable.: This defaults to shutil.which(‘crossbar’).

If data is None, returns None. Otherwise returns a CrossbarConfig object.

InstanceConfig

The structure of InstanceConfig encoding is described in the from_dict method of the InstanceConfig class:

class ocs.site_config.InstanceConfig[source]

classmethod from_dict(data, parent=None)[source]

Parameters:

data – The configuration dictionary.
parent – the HostConfig from which this data was extracted (this is stored as self.parent, but not used).

The configuration dictionary should have the following elements:

instance-id (str, required): This string is used to set the Agent instance’s base address. This may also be matched against the instance-id provided by the Agent instance, as a way of finding the right InstanceConfig.
agent-class (str, optional): Name of the Agent class. This may be matched against the agent_class name provided by the Agent instance, as a way of finding the right InstanceConfig.
arguments (list, optional):: A list of arguments that should be passed back to the agent.
manage (str, optional):: A string describing how a HostManager should manage this agent. See notes.

Notes

The manage value is only relevant if a HostManager is configured to operate on the host. In that case, the HostManager’s treatment of the agent instance depends on the value of manage:

“ignore”: HostManager will not attempt to manage the agent instance.
“host/up”: HostManager will manage the agent instance, launching it on the host system. On startup, the instance will be set to target_state “up” (i.e. the HostManager will try to start it).
“host/down”: like host/up, but HostManager will not start up the agent instance until explicitly requested to do.
“docker/up”: HostManager will manage the agent instance through Docker. On Startup, the instance will be set to target_state “up”.
“docker/down”: Like docker/up, but the instance will be forced to target_state “down” on startup.

In earlier versions of OCS, the acceptable values were “yes”, “no”, and “docker”. Those were equivalent to current values of “host/down”, “ignore”, and “docker/down”.

Those values are still accepted, but note that “yes” and “docker” are now equivalent to “host/up” and “docker/up”.

The following abbreviated values are also accepted:

“host”: same as “host/up”
“up”: same as “host/up”
“down”: same as “host/down”

Agent Site-related Command Line Parameters

Agent code can connect to the ocs.site_config using the following boilerplate code:

from ocs import site_config

class MyHardwareDevice:
    # ...

def make_parser(parser=None):
    if parser is None:
        parser = argparse.ArgumentParser()

    pgroup = parser.add_argument_group('Agent Options))
    pgroup.add_argument("--serial-number", help="Serial number of device")
    pgroup.add_argument("--mode", default="idle", choices=['idle', 'acq'])
    pgroup.add_argument('--num-channels', default=2, type=int,
                        help='Number of fake readout channels to produce. '
                        'Channels are co-sampled.')
    pgroup.add_argument('--sample-rate', default=9.5, type=float,
                        help='Frequency at which to produce data.')

if __name__ == '__main__':

    parser = make_parser()
    args = site_config.parse_args(agent_class='FakeDataAgent', parser=parser)

    print('I am in charge of device with serial number: %s' % args.serial_number)

    # Call launcher function (initiates connection to appropriate
    # WAMP hub and realm).
    agent, runner = ocs_agent.init_site_agent(args)

    my_hd = MyHardwareDevice()
    #... register processes and tasks, e.g.:
    agent.register_process('acq', my_hd.start_acq, my_hd.stop_acq)

    runner.run(agent, auto_reconnect=True)

The call to site_config.parse_args() will first pre-parse any command line arguments to determine the correct site, host, and instance to use for the specified agent_class. It will then merge any specified command line arguments with the instance arguments specified in the site-config file, and parse them with the full agent + site parser, so the returned namespace will contain additional site-info settable from the site’s parser.

usage:  [-h] [--site SITE] [--site-file SITE_FILE] [--site-host SITE_HOST]
        [--site-hub SITE_HUB] [--site-http SITE_HTTP]
        [--site-realm SITE_REALM] [--instance-id INSTANCE_ID]
        [--address-root ADDRESS_ROOT] [--registry-address REGISTRY_ADDRESS]
        [--log-dir LOG_DIR] [--working-dir WORKING_DIR]
        [--crossbar-timeout CROSSBAR_TIMEOUT] [--access-policy ACCESS_POLICY]

Site Config Options

--site: Instead of the default site, use the configuration for the specified site. The configuration is loaded from $OCS_CONFIG_DIR/{site}.yaml. If --site=none, the site_config facility will not be used at all.
--site-file: Instead of the default site config, use the specified file. Full path must be specified.
--site-host: Override the OCS determination of what host this instance is running on, and instead use the configuration for the indicated host.
--site-hub: Override the ocs hub url (wamp_server).
--site-http: Override the ocs hub http url (wamp_http).
--site-realm: Override the ocs hub realm (wamp_realm).
--instance-id: Look in the SCF for Agent-instance specific configuration options, and use those to launch the Agent.
--address-root: Override the site default address root.
--registry-address: Deprecated.
--log-dir: Set the logging directory.
--working-dir: Propagate the working directory.
--crossbar-timeout: Length of time in seconds that the Agent will try to reconnect to the crossbar server before shutting down. Note this is set per Agent in an instance’s arguments list.
--access-policy: Override Access Control policy.

Note

For examples calling these commandline arguments see Examples.