Rework accounting scripts

[anish-mudaraddi]

make accounting scripts run as a docker container

requirements:

1. the docker container must run continuously - to get accounting data every interval

2. also, the container can be used to collect past accounting data by providing timestamps from the past.

3. there should be good logs so it can run on kayobe and be debugged easily

4. there should be a dry-run feature - so we can test improvements in the future without sending data anywhere

5. there should be good testing and linting to keep up with our coding standards

6. it should work on a supported python version and the dependencies should be up-to-date

Still TODO:

• more testing on monitoring nodes
• acceptance testing on dev
• automated CI to run test
• moving this out into https://github.com/stfc/cloud-image-builders
  so that we can auto-build and version the docker container
• getting a playbook together in our kayobe config to install this alongside monitoring containers
• setting up perms on dev db to allow these scripts to run on monitoring nodes

[gmatthews20]

I haven't yet reviewed any of the tests. Will take a look once we're happy with the functionality

[anish-mudaraddi]

Changes made since last review:

using influxdb client library for talking to influxdb

• influxdb protocol is https - with self-signed certs - hence tls verify turned off

• removed --interval option - hardcoded to 24hours

• simplified --start-time and --end-time to just accept yyyy-mm-dd rather than full ISO timestamps - if omitted - start-time is hardcoded to be midnight of from the date it is run

• using influxdb python client library for Sink instead of requests library

  • using HTTPS protocol by default, with verify-tls as false as we use self-signed certs currently

• using env-vars for setting username, password, host for source and sink for added security - specify them in .env file

• updated readme

• fixed tests

• a bunch of linting fixes

• bumped the version to 1.1.0

• tested and working on staging kayobe

[DavidFair]

Partial re-review
Some minor bits just as I'm going through....

[DavidFair]

Full review - some security bits to validate and tighten up

Some doc / usability improvements where I noticed the version is mismatched in the README already and would need updating every time someone bumped this

[anish-mudaraddi]

changes since last review

• default --start-time is yesterday - so the full query + parsing + upload is run immediately, then the script waits 24 hours to parse the next interval

• pip install --no-cache-dir . is run as monitoring user and not root for added security in docker container

• set a global variable for hardcoded interval

• use version.txt to set the docker container version - I manually set the python package version in pyproject.toml - to fix in later PR - less so concerned with this as its intended to run inside the docker container and we're not shipping it as a standalone package

On leave