Get list of validation results for each rule/check?

[reinholdsson]

Hi,

Would it be possible to add support for returning (optionally) a dictionary or similar structure that maps each rule/check to its ValidationResult instance?

We have a use case for integrating dataframely's schema validation with Dagster asset checks. To enable this, we need a way to programmatically access each rule's validation result so we can dynamically map them to corresponding asset checks.

Currently, the validation output may look like:

ValidationError: 4 rules failed validation:
 - 'reasonable_bathroom_to_bedrooom_ratio' failed for 1 rows
 - 'minimum_zip_code_count' failed for 2 rows
 * Column 'zip_code' failed validation for 1 rules:
   - 'min_length' failed for 1 rows
 * Column 'num_bedrooms' failed validation for 1 rules:
   - 'nullability' failed for 2 rows

Hence, it would be great to have an API to obtain validation results in a structured mapping, e.g.:

{
    "reasonable_bathroom_to_bedrooom_ratio": ValidationResult(...),
    "minimum_zip_code_count": ValidationResult(...),
    "zip_code_min_length": ValidationResult(...),
    ...,
}

This would allow us to:

• Dynamically map each check to a Dagster asset check.
• Retrieve per-rule validation details programmatically, not just from the textual summary.

Thanks! :)

[AndreasAlbertQC]

Hey Thomas Falk (@reinholdsson) ! Good suggestion, I think we can definitely do a better job of exposing this info. As of today, you can:

1. run .filter instead of .validate whenever there is any chance of failure.
2. This returns a FilterResult, which contains a FailureInfo

FailureInfo contains the information you are looking for, albeit only in a private member _lf, which is a dataframe where the rows are the rows you filtered and the columns represent the checks (true = pass, false = fail, None=not evaluated). In our own internal code that uses dataframely, we have so far often converged on using this dataframe to infer the info we need. Clearly, that already hints we should either just make it public as is or enhance it a little and then make it public.
Oliver Borchert (@borchero) Daniel Elsner (@delsner) wdyt?

[reinholdsson]

Excellent, thanks!

I suppose MySchema.filter(df, cast=True).failure.counts() combined with MySchema._validation_rules(with_cast=True) gives me the info I need for my use case (although it would still be nice to have the validation results available as a public method).

[borchero]

We should definitely improve the usability of FailureInfo; there's been a bunch of questions/requests around this (see also #122, #124, #190). I think all of these issues should give us enough of an idea what common use cases look like and we should extend the public API accordingly.

[AndreasAlbertQC]

Hey Thomas Falk (@reinholdsson) : In #275, we have now exposed the per-row check information in FailureInfo.details(). Does that cover your use case?

[reinholdsson]

Andreas Albert (@AndreasAlbertQC) thanks you for looking into this, it's definitely an improvement. It doesn't exactly cover my use case at the moment since I only need to construct a dictionary with both the failed and passed validations (see example below), but it will be useful if/when I need to get more detailed validation results (e.g. counts of valid/invalid/unknown per rule). I got what I need now, so you may close this issue if you want, thanks!

rules = HouseSchema._validation_rules(with_cast=True)
failed_rules = HouseSchema.filter(df, cast=True, eager=True).failure.counts()
validation_results = {i: failed_rules.get(i) is None for i in rules}
validation_results

# => 
{
  'reasonable_bathroom_to_bedrooom_ratio': False,
  'num_bathrooms_ok': True,
  'zip_code|nullability': True,
  'zip_code|min_length': False,
  'num_bedrooms|nullability': False,
  'num_bedrooms|max': True,
  'num_bathrooms|nullability': True,
  'price|nullability': True,
  'price|inf': True,
  'price|nan': True,
  'zip_code|dtype': True,
  'num_bedrooms|dtype': True,
  'num_bathrooms|dtype': True,
  'price|dtype': True
}

Below is an example of how it's reported as asset checks within Dagster:

[AndreasAlbertQC]

Sweet, thanks for explaining Thomas Falk (@reinholdsson)! I am not very familiar with the details of dagster, but I wonder if the type of integration work you are doing would also be useful to others. I think it's a little too detailed to directly integrate new code into the dataframely package, but I think it would be cool to showcase this kind of integration in the docs. Would you be interested in contributing a docs page giving a simpe example of how to use dataframely with dagster? Assuming that this can be done with a reasonable amount of explainable code + a few screenshots, I think it would be neat.