Semantic Validation

Semantic vs Syntactic validations

Semantic validation and syntactic validation are two types of validation techniques used in software development to ensure that data entered into a system is accurate and conforms to the requirements of the system.

Syntactic validation checks the syntax of the input data and ensures that it adheres to the prescribed format. It checks whether the data entered is structured correctly and follows the expected syntax rules. For example, if an input field is supposed to accept only numerical data, a syntactic validation would ensure that only numerical characters are entered and reject any non-numeric characters.

Semantic validation, on the other hand, checks the meaning of the input data and ensures that it is valid in the context of the system. It checks whether the input data conforms to the business rules and logic of the system.

For example, if a system requires a date to be entered, a semantic validation would ensure that the date entered is valid, such as it’s not a future date or a date that has already passed.

In summary, syntactic validation checks the structure of the data, while semantic validation checks the meaning of the data. Both types of validation are important to ensure the accuracy and integrity of data entered into a system.

Introduction

Here we have created ‘Framework for semantic validation of observing setups’. This framework provides semantic validation which helps to prevent the users from making errors in their setups. This framework is supporting only for MID schema validation.

For creating this framework there are some requirements and architecture have already provided. These are as follows:

JSON validator file

A JSON file has been created to store all the parameters present in assign & configure resources along with its business rules and errors.

This JSON validator file has been imported in mid_oet_tmc_validators using below commmand:

tmdata/instrument/ska1_mid/validation

from ska_telmodel.data import TMData
tmdata = TMData()

validator_json_schema = tmdata[
    "instrument/ska1_mid/validation/mid-validation-constants.json"
].get_dict()

Adding a new parameter in JSON validator file

Steps to add a new parameter in JSON validator file

  • Locate the appropriate place in the JSON structure:
    • Identify the parent key or object where the new parameter should be added.

    • Determine the desired position for the new parameter within the parent key’s object.

  • Add a new key-value pair representing the parameter:
    • Structure of parameter should be parent-child.

    • Specify the name of the parameter as the key, this key represents the parent_key and it should contain dictionary.

    • Add additional key-value pairs within the parent_key object for the rule and error message. In this you can specify the business rule & error message to validate the specific key.

Example

If a user wants to add any new parameter in JSON validator file so he can take reference of this example:

"scan": {
        "tmc": {
            "scan_id": [
                {
                    "rule": "scan_id == 1",
                    "error": "Invalid input for scan_id"
                }
            ]
        }
    },

Let’s take scan command as a dummy key which is currently not present in the JSON file.

Here under scan there is a dictionary which has a key named “tmc” so scan.tmc will be the parent_key and under tmc we have a “scan_id” child key containing a list which should contain appropriate rules and error messages.

General structure

This framework has created very dynamically and user friendly. If user wants to access this framework from CDM or Jupyter Notebook then he just has to import telvalidation package from import statement and call semantic_validate function and pass the appropriate parameters to this function. If validation fails then the end user will get the list of errors.

This framework can be access by below command:

from ska_telmodel.telvalidation.semantic_validator import semantic_validate

There are some steps of this framework these are as follows:

  • Step 1

    It checks the parameter in the JSON validator document which is present in tmdata package.

  • Step 2

    There is a validate_json function which takes two parameters JSON file & config as a dictionary. It is present in src/ska_telmodel/telvalidation/mid_oet_tmc_validators. Here we are using an eval term to evaluate the business rules present in the JSON file and based on that it raises custom errors. All the custom errors are stored in a list named error_msg_list. At the end this function returns a list containing all the error messages.

    ska_telmodel.telvalidation.mid_oet_tmc_validators.validate_json(semantic_validate_constant_json: dict, command_input_json_config: dict, error_msg_list: list, parent_key: str) list[source]
    Parameters
    • semantic_validate_constant_json – json containing all the parameters along with its business semantic validation rules and error message.

    • command_input_json_config – dictionary containing details of the command input which needs validation. This is same as for ska_telmodel.schema.validate.

    • parent_key – temp key to store parent key, means if same semantic validation key present in 2 places this will help to identify correct parent.

    Returns

    error_msg_list: list containing all combined error which arises due to semantic validation.

  • Step 3

    There is one more function semantic_validate which takes mainly three argument config, interface and raise_semantic. It is present in src/ska_telmodel/telvalidation/schema.

    This function first checks for the interface, if the interface is not present then a warning message is logged, indicating that the interface is missing from the config. Additionally, a SchematicValidationError exception is raised with the same message.

    This framework allowed interface only for two commands that are assignresources & configure. If a user provides an incorrect or unsupported interface value, for example if user passes the interface for the scan command, the code will not be able to find a matching validation schema based on that interface. As a result, the validate_json function will not be called, and the msg_list variable will remain empty.

    Also this function is not supporting low telescope schema validation currently.

    ska_telmodel.telvalidation.semantic_validator.semantic_validate(config: dict, tm_data: ska_telmodel.data.frontend.TMData, interface: Optional[str] = None, raise_semantic: bool = True)[source]
    Parameters
    • config – dictionary containing details of the command which needs validation. This is same as for ska_telmodel.schema.validate. If command available as json string first convert to dictionary by json.loads.

    • tm_data – telemodel tm data object using which we can load semantic validate json.

    • interface – interface uri in full only provide if missing in config

    • raise_semantic – True(default) would need user to catch somewhere the SchematicValidationError. Set False to only log the error messages.

    Returns

    msg: if semantic validation fail returns error message containing all combined error which arises else returns True.

Target visibility validation

There are ra and dec parameters in configure resources, to validate these parameters we have created a separate module named coordinates_conversion which converts Right Ascension and Declination to Azimuth and Altitude. This module contains a function ra_dec_to_az_el which has logic for this conversion. This function has been imported in the validate_target_is_visible function which is present in the mid_oet_tmc_validators module.

ska_telmodel.telvalidation.mid_oet_tmc_validators.validate_target_is_visible(ra_str: str, dec_str: str, telescope: str, target_env: str, tm_data, observing_time: datetime.datetime = datetime.datetime(2023, 10, 20, 14, 14, 47, 480536)) str[source]

Check the target specific by ra,dec is visible during observing_time at telescope site

Parameters
  • ra_str – string containing value of ra

  • dec_str – string containing value of dec

  • telescope – string containing name of the telescope

  • observing_time – string containing value of observing_time

  • target_env – string containing the environment value(mid/low) for the target

  • tm_data – telemodel tm dataobject using which we can load semantic validate json.

This is the main function for conversion.

ska_telmodel.telvalidation.coordinates_conversion.ra_dec_to_az_el(telesc: str, ra: float, dec: float, obs_time: str, el_limit: float, tm_data: ska_telmodel.data.frontend.TMData, time_format: str = 'iso', if_set: bool = False, time_scale: str = 'utc', coord_frame: str = 'icrs', prec: float = 0.0001, max_iter: int = 200) list[source]
Returns

the az el in degrees from ra dec at given time for the telescopes [az el info_isvisible]

Index 0

azimuth in degrees

Index 1

elevation in degrees

Index 2

info_isvisible is True if src visible above/at el_limit given time else False

Parameters
  • telesc – “mid” for Mid or “low” for Low Telescope

  • ra – Right ascension in degrees with decimal places for arc min,arc sec also covert to degrees. Eg 123d30’ input 123.5 . In case of RA in hh mm sec please also convert to degrees.

  • dec – Declination in degrees with decimal places.

  • obs_time – str containing time when source position in terms of azimuth, elevation should be calculated. Eg ‘2023-04-18 20:12:18’

  • time_format – str to choose from available Time.FORMATS. Default “iso”

  • time_scale – str to choose from available Time.SCALES Default “utc”

  • coord_frame – str to choose from available Astronomical Coordinate Systems

  • el_limit – float specifying elevation in degree below which our telescope cannot observe the source

  • prec – float for precision limit in degrees to match elevation with given el_limit. default: 0.0001 degrees i.e. <1 arcsecond

  • max_iter – int to specify upto how many iterations can root finder use before it stops or reaches required precision. Default is 200. Only set higher if suggested by message. There is also a seperate message if it is determined that root finder is not able to converge starting from given time

  • tm_data – telemodel tm data object using which we can load semantic validate json.