Black Box Plugin

Purpose

The purpose of this plugin is to provide a "Black Box" recorder for the Ardexa edge device.

Usage

The Ardexa Black Box recorder, as the name suggests, records all relevant Ardexa edge environmental factors, to the Ardexa cloud, at regular intervals. Normally, the ArdexaLinux image contains all the necessary modules to manage the black box recorder. This plugin is an update to the ArdexaLinux Black Box module. As of Version 1.6.5 or greater, data will be sent to the cloud and be available in the ANALYSIS section of the Ardexa cloud. This plugin is scheduled to run every 5 minutes by the native Linux Cron subsystem. The following data will be recorded, noting that some data may not be available on certain hardware platforms:

Current hour:  14
Current minute:  26
  Table                                               black_box       
  Source                                              Black box       
  CPU usage(decimal:%)                                34.3            
  MEM usage(decimal:%)                                22.1            
  SWAP usage(decimal:%)                               0.1             
  DISK usage(decimal:%)                               24.3            
  Ardexa Broker DNS ping(keyword)                     good            
  8_8_8_8 IP ping(keyword)                            no attempt      
  Gateway ping(keyword)                               no attempt      
  NS Lookup broker_ardexa_com using 8_8_8_8(keyword)  no attempt      
  Ardexa running(keyword)                             True            
  Agent run state(keyword)                            True            
  Agent cloud connection(keyword)                     True            
  Agent message count(integer)                        126435          
  Agent loop count(integer)                           1               
  Agent core loop state(keyword)                      True            
  Agent comm loop state(keyword)                      True            
  Agent core ticker(integer)                          667703          
  Agent comm ticker(integer)                          801399          
  Agent message cache(integer)                        0               
  Timezone(keyword)                                   Australia/Sydney
  CPU temp(decimal:°C)                                55.8            
  Uptime(decimal:h)                                   2906.62         
  Ardexa PID(integer)                                 3803 
  Delete data(keyword)                                False
  5 Minute Weighted Load Average (decimal)             False

The 5 Minute Weighted Load Average is the load average over 5 minutes, divided by the number of cores/CPUs. In addition, the following data is dumped to a file for later forensics. This "dump" file is located at /var/log/ardexa-black-box-dumps.log, and will record data as follows:

1. if CPU exceeds 60%, it will dump the top 4 processes by CPU usage

2. if MEM exceeds 60% or SWAP by 15%, it will dump the top 4 processes by MEM usage

3. all connected USB devices are dumped every 5 minutes

4. once a day, the contents of /etc/resolv.conf is dumped

The dump files are archived on a daily basis and kept for about 3 months. Dump files older than 3 months are deleted.

If the black box detects that more than 75% of the disk has been used, it will attempt to delete data from the logging area (/opt/ardexa/logs). If this area uses less than 5% of the disk, data will not be deleted. Only the oldest month/year will be deleted in a 5 minute run.

Also; black box will stop entries from journald being sent to syslog. This is accomplished by setting ForwardToSyslog=no in the /etc/systemd/journald.conf file.

Troubleshooting

Follows these steps to troubleshoot this plugin

• Check that version 2.0.0 or greater is installed, as follows:

pip3 list | grep ardexa

• The command ardexa_black_box will run the black box and log data to the latest file. Check it is being received by running the command: cat /opt/ardexa/logs/black_box/Black\ box/latest.csv

• Ensure an entry in the /etc/cron.d directory, under a filename called ardexa-black-box. Cron will be used to run the 5 minute runs. The Ardexa agent is NOT used for the scheduled run.