Ardexa Control Plugin

Overview

The control-ardexa plugin is able to undertake simple (non feedback) control of modbus and OPCUA enabled devices. In broad terms, the control plugin undertakes these features:

It will set one or more outputs based on one or more prioritised inputs.
The inputs are checked by a main program thread. The inputs can either be a file, OPCUA or Modbus based inputs.
Each output is controlled by each own thread. Outputs can be Sunny TriPower 110-60, Sunspec, Aurora or Bluelog Logger.
The poll rate, at which the input(s) are read is set by a control file. The control file is formatted as a YAML file.
The outputs will be read at the poll rate, or at multiples of the poll rates (eg; every 3rd poll rate)
All activity is logged to the control table in the cloud, and errors are sent to the control-errors table.

The plugin will only run as a service. Only one instance of the control_ardexa binary can run at any one time. This is controlled by a lock file.
Ideally, the control plugin can be run as any Linux user. However, for security pruposes, Ardexa will run the control plugin as root.

Activating

The control plugin is installed via the normal plugin installation process. It is activated via Ardexa Front End. In doing so, remember the following:

One 1 instance of the plugin should be implemented. Do not enable any more than 1.
Control of the plugin is enabled by the control file. The Ardexa Front End will not be used (for now) to configure the control file.
Once a new control file is uploaded, the plugin will detect this new file and immediately restart itself. There is no need to issue a service restart. The file must be called control.yaml. Use the upload entry in the Ardexa Front end to send the new file.
The frequency in the scenario at the Ardexa Front End is ignored. You can enter any value.

Control File

An example control file is shown below

# comments are ignored
# Scenario

poll_rate: 800
read_rate: 2
attempts: 2
debug: 0
mode: on

Scenario: 
  1:
    inputs:     
      1:
        endpoint: /opt/ardexa/config/file_input.yaml
        priority: 1
        type: FILE

      2:
        endpoint: 127.0.0.1
        port: 1401
        priority: 2
        type: MoxaE1214
        # Defined as "Modbus value: Percent Setting"
        setpoint_values:
          0 : 100
          1 : 0
          2 : 30
          4 : 60

    outputs:
      3:
        endpoint: 192.168.7.11
        port: 502
        type: SMASTP110
        control: percent_active_power
        address: 3
        timeout: 455
      4:
        endpoint: /dev/ttyS0
        type: aurora
        address: 3
        control: percent_active_power
        timeout: 1780

  2:
    inputs:     
      1:
        endpoint: /opt/ardexa/config/file_input2.yaml
        priority: 1
        type: FILE

      2:
        endpoint: 128.0.0.1
        port: 1401
        priority: 2
        type: MoxaE1214
        # Defined as "Modbus value: Percent Setting"
        setpoint_values:
          0 : 100
          1 : 0
          2 : 30
          4 : 60

    outputs:
      3:
        endpoint: 128.0.0.1
        port: 1610
        type: BlueLog
        control: percent_active_power
        address: 1
        timeout: 1000

      4:
        endpoint: 128.0.0.1
        port: 1611
        type: BlueLog
        control: percent_active_power
        address: 10

The Global Settings applies to all inputs and outputs of the control plugin. These global settings are defined in the control file at the same level as the scenarios key, are as follows. If they are not defined in the control file, the default values specified below will be used

poll_rate: This is a value in milliseconds. It defines the rate at which all the inputs are read. The poll rate maximum (slowest) is 120,000 milliseconds (2 minutes). The fastest can be 0 milliseconds - although this is no recommended. If not defined, the poll rate is defaulted to 1000 milliseconds. Make sure the poll rate is set to be a value higher than the normal response rate of the slowest output. Also; Do not set a poll rate that is significantly quicker than what is required. This will simply waste resources
read_rate: This is a value that defines how often to read the outputs. A read rate of 1 means read the outputs every poll rate. Whereas a value of (say) 3 means read the outputs every 3rd poll rate. When there is a change to the input, the outputs will be changed as soon as possible but no later than the poll rate, that is; when an input is changed and an output needs to be changed to meet this new input, the read_rate is ignored. The read rate maximum is 100. If not defined, the read rate is defaulted to 1. Do not set a read rate that is significantly quicker than what is required. This will simply waste resources and will interfere with normal collection Outputs do not need to be read at very frequent intervals. The plugin will check and reset an inverter if it does not meet the required setpoint.
attempts: This is a value that defines how many attempts to try and read the outputs (not the inputs). The attempts maximum is 100. If not defined, the attempts is defaulted to 2.
debug: This is a value of 0 (no Debugging), 1 (some debugging) or 2 (extensive debugging). It defines the debugging level. Debug entries are not sent to the cloud due to the amount of logs they can generate. However; errors and alerts are sent to the cloud (read below). If not defined, the debug is defaulted to 0. Only change the debug level when directed by the Ardexa Development team. Make sure the debug entry is turned off (to 0) when debugging is no longer required
mode: This can be defined as On, Off or NoConfig. In normal operation, this should be defined to on. In off or NoConfig, the plugin will not read any inputs or outputs. It will continue to poll at the poll rate, and log audit entries. If not defined, the mode is defaulted to on.

Scenarios. The YAML file example show 2 scenarios. Any number of scenarios can be defined in the YAML file. They must have their own unique number. Each input and output within a scenario must also have its own unique number. These numbers are used for logging and debugging, and identify the threads. Inputs or outputs cannot be shared with other scenarios. That is to say, any endpoint/port pair in any single scenario, cannot appear again in that scenario, OR another scenario.
Inputs. Each input must have a unique endpoint, port and priority number. The lowest number means the highest priority, and so the highest priority is "1". Allowed input type include:

a.  MOXA E1214. To use this input include the string "moxae1214" in the config file.
b.  FILE. This is a standard file input. To use this input include the string "file" in the config file.

Each MOXA must have a setpoint_values defined that maps the register 48 value to a percentage figure. The map must use integers only. Each input can have a timeout defined, in milliseconds. This timeout is the maximum time the plugin will wait for the input to respond. If it responds earlier than this time, then the plugin will not wait for the maximum time, it will return as soon as the data is read. The default timeout value if not defined is 1000.

Outputs: Each output must have a unique endpoint, port. Allowed output type include:

Bluelog Logger. To use this output include the string bluelog in the config file. The Bluelog logger only allows setting of active power as a percentage value. The Ardexa Control plugin will set the watchdog on the logger. Bluelog logger can only be contacted via TCP, so serial endpoint will not be accepted.
Sunspec enabled solar inverters. To use this output include the string sunspec in the config file. Sunspec inverters only allows setting of active power as a percentage value. For now, only connections via TCP will be accepted. Serial endpoint will not be accepted.
Sunny TriPower STP 110-60. To use this output type include the string SMASTP110 in the config file. Sunny TriPower STP 110-60 inverters only allows setting of active power as a percentage value. For now, only connections via TCP will be accepted. Serial endpoint will not be accepted. Also note that the Ardexa Control plugin WILL NOT write values to EEPROM (only RAM memory) and in any case will only write values to the inverter when it is absolutely required to meet a setpoint command. From the SMA documentation:

Except of model 123 (Immediate Inverter Controls) all data is written to flash memory - i.e. excessive write access can damage the device. As the data in model 123 is stored in RAM, it does not get restored after inverter shutdown. Cyclical writing is required.

ABB Aurora. To use this output type include the string aurora in the config file. Aurora inverters only allows setting of active power as a percentage value. Connections via TCP or serial aree accepted. Aurora uses address and so an address entry must be included, or a default of 1 will be used. The plugin will ask the inverter to respond to a change in setpoint within 1.2 seconds of the command being issued. The Ardexa Control plugin WILL NOT write values to EEPROM (only RAM memory) and in any case will only write values to the inverter when it is absolutely required to meet a setpoint command. From the ABB documentation:

...Due to the fact the allowed number of writings on EEPROM is limited, this method shall not be used in case of dynamic control methods....

Each output must also have a control defined, which for now can only be percent_active_power. Each output can have a timeout defined, in milliseconds. This timeout is the maximum time the plugin will wait for the output to respond. If it responds earlier than this time, then the plugin will not wait for the maximum time, it will return as soon as the data is read. The default timeout value if not defined is 1000. For outputs that use Modbus (including Sunspec), an address value may be defined. The default is 1, if an address entry is not included with the output. It can be less than 1 nor greater than 255. An address included for outputs that do not use them will be ignored.

If an output setpoint has been set, and the inverter cannot subsequently be contacted for whatever reason, the control plugin will continue to attempt to access the inverter. The previous successful setpoint set on the inverter will be assumed to still be current on the inverter.

If, for any reason, they control file cannot be read, the control plugin will go into a simple loop state; where it will not monitor any inputs or outputs. The audit trail will log an entry as NoConfig every 60 seconds, if a valid YAML file is not found.

Control File Testing and Backups

The control plugin will automatically save a copy of each unique copy of the control file. These files will have a name something like: 2AA4C5EE_2024-09-11T14:49:08+10:00_config_file.yaml. The checksum is the leading string 2AA4C5EE. This checkum will appear in the audit logs (see below) and uniquely identifies the control file being used. Each time a new configuration file is copied to the control plugin, it will be copied to the archive area (/opt/ardexa/config/control-ardexa/config_file_backups/) ONLY IF the checksum does not already exist. The config file backups will be stored for the life of the system on the Ardexa cloud.
Config files are checked for errors will not be copied to the backup area. Checking for errors can be undertaken by running the command: control_ardexa -c {CONFIG FILE LOCATION"} -y for example; control_ardexa -y -c /tmp/location/control.yaml

Discovery

The operation of the plugin can be determined by running the DISCOVER button in the Ardexa front end. On the command line, this is acheived by running the command control_plugin -d

Audit, Error and Debugging Logs

There are 3 levels of reporting enabled on the control plugin, audit logging, error logging and debug logging.
audit logs are generated once every 60 seconds and whenever any setpoint is changed. In either of these conditions, a log entry will be made for every input and output. Each log entry will be sent to the cloud and appear in the control table. Audit logs contain the following fields:

datetime: date              ... This is the datetime of the audit log
scenario_id: String         ... This is the scenario ID. Something like: "1"
thread_id: String           ... This is the thread ID. Something like: "1.3" ("1" is the parent scenario ID)
mode: ModeType              ... Will "On", "Off" or "NoConfig"
endpoint: String            ... This will be the endpoint string. Something like "127.0.0.1"
port: String                ... This is the TCP port (if it exists). Something like "502"
endpoint_type: String       ... This is the type of endpoint. It will be either "BlueLog", "SMASTP110" or "Sunspec"
control_type: String        ... If the log record refers to an output it will one of "ActivePower", "ReactivePower" or "PowerFactor". Otherwise "Input" will appear
action: LogAction           ... This will be one of "ChangeSetpoint", "RegularReport" (that is; a 60 second report), "SetPointChanged", "SetPointNotChanged" or "YAMLChanged"
current_setpoint: String    ... The current setpoint for the output. It may be empty if no setpoint defined
config_checksum: String     ... This is the CRC32 Checksum of the config file
process_id: String          ... The process ID of the control plugin. Used to track the control plugin reliability
contactable: bool           ... "true" if the output is contactable

error logs are recorded whenever an error condition (one that causes the control plugin to stop), or an alert condition (a critical condition that does NOT result in program termination) occurs. These records are sent to the cloud, and appear in the control-errors table. ERROR logs indicate a fatal program condition resulting in program termination. In such cases, the control plugin will be automatically restarted. Error logs contain the following fields:

datetime: date              ... This is the datetime of the error log
Error Code: String          ... This will be a code like "ERROR23". This will provide the developers are key to where the condition occurred in the program
Description: String         ... This is a full description of the error

debug logs will be recorded if the YAML file contains an entry of debug: 1 or debug: 2. Debugging will generate a huge amount of logs, and will not be sent to the cloud. The logs will be written to the local machine, on the file: /var/log/ardexa-control.log. These logs are used by developers to fault find critical issues, and so must only be set to 1 or 2 on direction of the development staff. Debugging generates a large volume of logs, and so therefore .. DO NOT leave debugging on more than is required.

Troubleshooting

The following steps must be undertaken to troubleshoot the control plugin. When reporting any issues, send all the following data to the Ardexa development team.

Check the status of the control plugin service and make sure it is running: sudo systemctl status control-ardexa.service
If the process is not running, check PID file: cat /var/lock/ardexa_control.pid. The contents of this file is the process ID. Check the process ID is running as follows: ps -p {PID}. It should return something like:

    PID TTY          TIME CMD
 440271 ?        00:01:41 control_ardexa

If the process is not running, remove the PID file as follows: rm /var/lock/ardexa_control.pid and restart the service as follows: sudo systemctl restart control-ardexa.service

Check the version of the control plugin: control_ardexa --version. If its not the latest version, upgrade it.
Check the control file does not have any errors: control_ardexa -y. If you have the control file in another temporary location, use: control_ardexa -y -c /tmp/location/control.yaml
Check the audit log: cat /opt/ardexa/logs/control/control-ardexa/latest.csv, and make sure an entry is being recorded at least every 60 seconds
Check the error log for any obvious errors: cat /opt/ardexa/logs/control-errors/control-ardexa/latest.csv
If debug is turned on, check the debug log: cat /var/log/ardexa-control.log
Check to see if the control plugin is able to discover data: control_ardexa -d

PreviousControl NextVariable Naming Guide