4 - Develop

The basic API interaction is handled in ‘ansibleguy.opnsense.plugins.module_utils.base.api’.

It is a generic abstraction layer for interacting with the api - therefore all plugins should be able to function with it!

Install

You can install the collection to a specific directory for easier testing.

cd $PLAYBOOK_DIR
ansible-galaxy collection install git+https://github.com/ansibleguy/collection_opnsense.git,<COMMIT/BRANCH> -p ./collections

Of course you can always place the repository at ${PLAYBOOK_DIR}/ansible_collections/ansibleguy/opnsense so it gets picked-up by Ansible.

API Definition

To get to know the API - you will have to read into the API’s XML-config that is linked in the OPNSense docs.

Per example: Alias.xml

As XML isn’t the most readable format - I would recommend translating it to YAML or JSON.

Here is a nice online-tool to do so: XML-to-YAML | XML-to-JSON

Module

There are module-templates that can be copied - so you don’t have to re-write the basic structure.

Abstraction

  • Module Abstraction

    img_module_abstraction

  • Module Action-Abstraction

    img_module_abstraction_actions

Adding new module

Checklist:

  • Create the module-file at:

    ‘<COLLECTION>/plugins/modules/<MODULE>.py’

    You can copy the template from ‘<COLLECTION>/plugins/modules/_tmpl_obj.py’

    Note: When adding module-parameters - you can copy/paste the field-description from the OPNSense web-ui! We don’t have to reinvent the wheel. (‘full help’ toggle)

  • For most modules you should create a sub-file to handle the actual logic so the main module-file is kept clean:

    ‘<COLLECTION>/plugins/module_utils/main/<MODULE>.py’

    You can copy the template from ‘<COLLECTION>/plugins/module_utils/main/_tmpl.py’

  • Add ansible-based tests:

    I personally like to write tests before adding new modules and testing the modules functionality from the start (test-driven-development)

    • You can copy the template from ‘<COLLECTION>/tests/_tmpl.yml’

      Rename all calls to the new module.

    • Add a cleanup-task in ‘<COLLECTION>/tests/cleanup.yml’ (set state we will expect when re-running the tests)

    • Enable the test once it runs successfully - add it to ‘<COLLECTION>/scripts/test.sh’

  • Add documentation:

    • You can copy the template from ‘<COLLECTION>/docs/source/_tmpl/module_template.rst’ and replace ‘<module>’ and links

      reStructuredText is preferred, but markdown is also supported

      Also add important module-specific information.

    • Optional: We should also add inline module-documentation as standardized for Ansible

      To keep the main module file clean - the documentation should be placed in ‘<COLLECTION>/plugins/module_utils/inline_docs/’

      You can copy the template from ‘<COLLECTION>/plugins/module_utils/inline_docs/_tmpl.py’

      You can then import the documentation inside the main module file.

  • Add the module to ‘<COLLECTION>/meta/runtime.yml’

  • Add the module as option to the ‘ansibleguy.opnsense.list’ module:

    ‘<COLLECTION>/plugins/modules/list.py’

  • Add the module as option to the ‘ansibleguy.opnsense.reload’ module:

    ‘<COLLECTION>/plugins/modules/reload.py’

  • If you are implementing a new service:

    Add the service as option to the ‘ansibleguy.opnsense.service’ module:

    ‘<COLLECTION>/plugins/modules/service.py’

Testing

Run the tests like this:

# set these variables:
COL='name-of-new-collection'
COL_PATH="$(pwd)/../collections/ansible_collections/ansibleguy/opnsense"  # path to your local collection
TEST_FIREWALL='192.168.0.1'  # ip of your test-firewall
TEST_API_KEY="$(pwd)/opn.txt"  # api credentials-file for your test-firewall
export ANSIBLE_DIFF_ALWAYS=yes  # enable diff-mode for debugging

bash "${COL_PATH}/scripts/test_single.sh" "$TEST_FIREWALL" "$TEST_API_KEY" "$COL_PATH" "$COL" 1

API

One can choose to either:

  • create a http-session - faster if multiple calls are needed

    p.e. check current state => create/update/delete

    from ansible_collections.ansibleguy.opnsense.plugins.module_utils.base.api import Session
    session = Session(module=module)
    session.get(cnf={'controller': 'alias', 'command': 'addItem', 'data': {'name': 'dummy', ...}})
    session.post(cnf={'controller': 'alias', 'command': 'delItem', 'params': [uuid]})
    session.close()
    

    or using a context-manager:

    from ansible_collections.ansibleguy.opnsense.plugins.module_utils.base.api import Session
    with Session(module=module) as session:
        session.get(cnf={'controller': 'alias', 'command': 'addItem', 'data': {'name': 'dummy', ...}})
        session.post(cnf={'controller': 'alias', 'command': 'delItem', 'params': [uuid]})
    
  • use a single call - if only one is needed

    p.e. toggle a cronjob or restart a service

    from ansible_collections.ansibleguy.opnsense.plugins.module_utils.base.api import single_get, single_post
    single_get(
        module=module,
        cnf={'controller': 'alias', 'command': 'addItem', 'data': {'name': 'dummy', ...}}
    )
    single_post(
        module=module,
        cnf={'controller': 'alias', 'command': 'delItem', 'params': [uuid]}
    )
    

For the controller/command/params/data definition - check the OPNSense API Docs!

Debugging

Verbose output

If you want to output something to ansible’s runtime - use ‘module.warn’:

NOTE: This output is buffered by Ansible until the task has finished.

module.warn(f"{before} != {after}")

You can also use the debug argument to enable verbose output of the api requests.

- name: Example
  ansibleguy.opnsense.alias:
    debug: true

‘Multi’ modules also support the debug parameter on a per-item basis - so you don’t get flooded.

When the debug-mode is enabled some useful log files are created in the directory /tmp/ansibleguy.opnsense

guy$ ls -l /tmp/ansibleguy.opnsense/
alias.log  # time consumption profiling for the executed module: https://docs.python.org/3/library/profile.html
api_calls.log  # a list api calls that were executed by the debugged module

Profiling

To profile a modules time-consumption - you can use the existing profiler function:

For it to work, you need to move your modules processing into a dedicated function or object!

The profiler will wrap around this function call and analyze it.

from ansible_collections.ansibleguy.opnsense.plugins.module_utils.utils import profiler
from ansible_collections.ansibleguy.opnsense.plugins.module_utils.target_module import process

if module.params['profiling']:
    profiler(
        check=process, kwargs=dict(
            m=module, p=module.params, r=result,
        ),
    )

else:
    process(m=module, p=module.params, r=result)

Note: these entries can be interpreted as waiting for the responses of HTTP requests:

  • ‘read’ of ‘_ssl._SSLSocket’

  • ‘connect’ of ‘_socket.socket’

  • ‘do_handshake’ of ‘_ssl._SSLSocket’

One can only try to lower the needed HTTP calls.