Quality filtering

Martin Westgate

2026-11-20

What is ‘data quality’?

Databases containing biodiversity information - such as those that can be queried using galah - contain millions of records collected over decades or centuries by thousands of observers. It is unsurprising, therefore, that they contain some errors. In this context, two key questions for users of that data are:

Data quality systems in galah are built on a set of fields called ‘assertions’. These are usually binary (true/false) variables that flag that something might be wrong with the data, generated either programmatically or by users. You can see them by calling show_all_assertions():

## # A tibble: 124 × 4
##    id                                 description                        category type      
##    <chr>                              <chr>                              <chr>    <chr>     
##  1 AMBIGUOUS_COLLECTION               Ambiguous collection               Comment  assertions
##  2 AMBIGUOUS_INSTITUTION              Ambiguous institution              Comment  assertions
##  3 BASIS_OF_RECORD_INVALID            Basis of record badly formed       Warning  assertions
##  4 biosecurityIssue                   Biosecurity issue                  Error    assertions
##  5 COLLECTION_MATCH_FUZZY             Collection match fuzzy             Comment  assertions
##  6 COLLECTION_MATCH_NONE              Collection not matched             Comment  assertions
##  7 CONTINENT_COORDINATE_MISMATCH      continent coordinate mismatch      Warning  assertions
##  8 CONTINENT_COUNTRY_MISMATCH         Continent country mismatch         Warning  assertions
##  9 CONTINENT_DERIVED_FROM_COORDINATES Continent derived from coordinates Warning  assertions
## 10 CONTINENT_DERIVED_FROM_COUNTRY     continent derived from country     Warning  assertions
## # ℹ 114 more rows

The easiest thing to do with these is simply download them with your chosen records, and do your own checking on the data. This can be achived via select():

galah_call() |>
  identify("Eolophus roseicapilla") |>
  filter(year >= 2020,
         basisOfRecord == "PRESERVED_SPECIMEN") |>
  select(group = c("basic", "assertions")) |>
  collect()
## Retrying in 1 seconds.
## Retrying in 2 seconds.
## Retrying in 4 seconds.
## Retrying in 8 seconds.
## # A tibble: 42 × 133
##    recordID              scientificName taxonConceptID decimalLatitude decimalLongitude eventDate           basisOfRecord occurrenceStatus dataResourceName AMBIGUOUS_COLLECTION AMBIGUOUS_INSTITUTION
##    <chr>                 <chr>          <chr>                    <dbl>            <dbl> <dttm>              <chr>         <chr>            <chr>            <lgl>                <lgl>                
##  1 03e1c487-f3be-4469-a… Eolophus rose… https://biodi…           -34.8             148. 2022-01-14 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  2 0a24639a-1387-4a53-b… Eolophus rose… https://biodi…           -34.8             148. 2022-01-14 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  3 0eaf52fc-6659-41ea-8… Eolophus rose… https://biodi…           -24.2             139. 2026-08-26 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  4 10081d36-abf8-4928-9… Eolophus rose… https://biodi…           -33.8             151. 2024-12-11 00:00:00 PRESERVED_SP… PRESENT          Australian Muse… FALSE                FALSE                
##  5 10e9d3c3-1fb2-4362-8… Eolophus rose… https://biodi…           -33.6             151. NA                  PRESERVED_SP… PRESENT          Australian Muse… FALSE                FALSE                
##  6 11663284-5f43-4cf2-a… Eolophus rose… https://biodi…           -28.2             144. 2025-09-04 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  7 1baea6d4-a2d1-4e69-8… Eolophus rose… https://biodi…           -36.4             144. 2024-09-21 00:00:00 PRESERVED_SP… PRESENT          Museums Victori… FALSE                FALSE                
##  8 1fe45f99-a245-44d7-b… Eolophus rose… https://biodi…           -28.3             136. 2025-08-02 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
##  9 3810878b-8ad7-4a86-a… Eolophus rose… https://biodi…           -32.6             150. 2022-05-14 00:00:00 PRESERVED_SP… PRESENT          Australian Muse… FALSE                FALSE                
## 10 3e6152db-aef1-4e32-b… Eolophus rose… https://biodi…           -30.1             138. 2025-08-01 00:00:00 PRESERVED_SP… PRESENT          Australian Nati… FALSE                FALSE                
## # ℹ 32 more rows
## # ℹ 122 more variables: BASIS_OF_RECORD_INVALID <lgl>, biosecurityIssue <lgl>, COLLECTION_MATCH_FUZZY <lgl>, COLLECTION_MATCH_NONE <lgl>, CONTINENT_COORDINATE_MISMATCH <lgl>,
## #   CONTINENT_COUNTRY_MISMATCH <lgl>, CONTINENT_DERIVED_FROM_COORDINATES <lgl>, CONTINENT_DERIVED_FROM_COUNTRY <lgl>, CONTINENT_INVALID <lgl>, COORDINATE_INVALID <lgl>,
## #   COORDINATE_OUT_OF_RANGE <lgl>, COORDINATE_PRECISION_INVALID <lgl>, COORDINATE_REPROJECTED <lgl>, COORDINATE_REPROJECTION_FAILED <lgl>, COORDINATE_REPROJECTION_SUSPICIOUS <lgl>,
## #   COORDINATE_ROUNDED <lgl>, COORDINATE_UNCERTAINTY_METERS_INVALID <lgl>, COORDINATES_CENTRE_OF_COUNTRY <lgl>, COORDINATES_CENTRE_OF_STATEPROVINCE <lgl>, COUNTRY_COORDINATE_MISMATCH <lgl>,
## #   COUNTRY_DERIVED_FROM_COORDINATES <lgl>, COUNTRY_INVALID <lgl>, COUNTRY_MISMATCH <lgl>, DEPTH_MIN_MAX_SWAPPED <lgl>, DEPTH_NON_NUMERIC <lgl>, DEPTH_NOT_METRIC <lgl>, DEPTH_UNLIKELY <lgl>,
## #   detectedOutlier <lgl>, DIFFERENT_OWNER_INSTITUTION <lgl>, ELEVATION_MIN_MAX_SWAPPED <lgl>, ELEVATION_NON_NUMERIC <lgl>, ELEVATION_NOT_METRIC <lgl>, ELEVATION_UNLIKELY <lgl>, …

As you can see, there are a huge number of assertions, and they will take some processing. It is also inefficient to download data we don’t need, only to filter it later. Therefore, it is worth considering filtering out some problematic records as part of our usual filtering strategy.

Data quality profiles

Data quality ‘profiles’ are pre-populated filter statements curated by the ALA that are designed to remove problematic records. They provide a quick and straightforward way to remove these records without having to construct elaborate filter statements.

show_all_profiles()
## # A tibble: 5 × 4
##      id short_name name                                  description                                                                                                                                  
##   <int> <chr>      <chr>                                 <chr>                                                                                                                                        
## 1    92 ALA        ALA General                           "The default ALA profile filters out records based on the filter groups outlined below.  This is a moderately restricted set of data."       
## 2   124 CSDM       Species Distribution Modelling (CSDM) "Base filters for the Collaborative Species Distribution Modelling program"                                                                  
## 3   133 re-usable  Data licensed for all uses            "Data licensed for re-use, including commercial uses. This profile only filters on license, no \"quality\" filters are applied."             
## 4   224 AVH        AVH                                   "AVH data quality profile"                                                                                                                   
## 5   252 ALA-test   ALA General Test                      "A test version of the default ALA profile filters out records based on the filter groups outlined below.  This is a moderately restricted s…

Let’s take a look at the most obvious one first:

search_all(profiles, "CSDM") |>
  show_values()
## • Showing values for 'CSDM'.
## # A tibble: 4 × 4
##      id description                                                               filter                        enabled
##   <int> <chr>                                                                     <chr>                         <lgl>  
## 1   126 "Include only records where Spatial validity is \"true\""                 "spatiallyValid:\"true\""     TRUE   
## 2   128 "Exclude potential duplicate records"                                     "-isDuplicateOf:*"            TRUE   
## 3   130 "Exclude all records that are an outlier against any environmental layer" "-outlierLayerCount:[1 TO *]" TRUE   
## 4   132 "Include only records where Year is 1970 to 2099"                         "year:[1970 TO *]"            TRUE

This is ok, but needs unpacking:

Now let’s look at the ‘ALA general’ approach:

 search_all(profiles, "ALA") |>
  show_values()
## ! Search returned 2 matched profiles.
## • Showing values for 'ALA'.
## # A tibble: 26 × 4
##       id description                                                                                                                                                                    filter enabled
##    <int> <chr>                                                                                                                                                                          <chr>  <lgl>  
##  1    94 "Exclude all records where spatial validity is \"false\""                                                                                                                      "-spa… TRUE   
##  2    96 "Exclude all records with an assertion that the scientific name provided does not match any of the names lists used by the ALA.  For a full explanation of the ALA name match… "-ass… TRUE   
##  3    97 "Exclude all records with an assertion that the scientific name provided is not structured as a valid scientific name. Also catches rank values or values such as \"UNKNOWN\"" "-ass… TRUE   
##  4    99 "Exclude all records with an assertion that kingdom provided doesn't match a known kingdom e.g. Animalia, Plantae"                                                             "-ass… TRUE   
##  5   100 "Exclude all records with an assertion that the scientific name provided in the record does not match the expected taxonomic scope of the resource e.g. Mammal records attrib… "-ass… TRUE   
##  6    98 "Exclude all records with an assertion that the name and classification supplied can't be used to choose between 2 homonyms"                                                   "-ass… TRUE   
##  7   103 "Exclude all records with an assertion of latitude value provided is zero"                                                                                                     "-dec… TRUE   
##  8   104 "Exclude all records with an assertion of longitude value provided is zero"                                                                                                    "-dec… TRUE   
##  9   105 "Exclude all records with an assertion of  latitude and longitude have been transposed"                                                                                        "-ass… TRUE   
## 10   106 "Exclude all records with an assertion of coordinates are the exact centre of the state or territory"                                                                          "-ass… TRUE   
## # ℹ 16 more rows

There is a lot here and we won’t go through it all; but two points are potentially useful. First, this makes a lot of assumptions about what is and is not useful, with 26 separate filters. This is a lot, and while most of them are useful in some contexts, equally this may be too stringent a set of tests for some applications. Applying it to our earlier example, we see it only removes two records, but your milage will vary for different queries:

galah_call() |>
  identify("Eolophus roseicapilla") |>
  filter(year >= 2020,
         basisOfRecord == "PRESERVED_SPECIMEN") |>
  apply_profile("ALA") |>
  count() |>
  collect()
## # A tibble: 1 × 1
##   count
##   <int>
## 1    40

Second, the filter column lists how the query is sent to the ALA; this can be useful for replicating parts of the workflow that are useful for your specific application. We’ll try this out below.

Bespoke filters using assertions

Assertions are stored in what is called a ‘multi-value field’, meaning a column where each cell can contain multiple strings. You remove them using the ‘assertions’ field name in filter:

galah_call() |>
  filter(assertions != "INVALID_SCIENTIFIC_NAME") |>
  count() |>
  collect()
## # A tibble: 1 × 1
##       count
##       <int>
## 1 195446531

Conversely, if you only want records with an invalid name, just use ==:

galah_call() |>
  filter(assertions == "INVALID_SCIENTIFIC_NAME") |>
  count() |>
  collect()
## # A tibble: 1 × 1
##     count
##     <int>
## 1 3620270

Of course, we might want to filter out multiple assertions at once. Rather than keep writing assertions !=, we can use %in% with a leading ! (meaning NOT):

excluded_assertions <- c("INVALID_SCIENTIFIC_NAME", 
                         "COORDINATES_CENTRE_OF_STATEPROVINCE")
galah_call() |>
  filter(!(assertions %in% excluded_assertions)) |>
  count() |>
  collect()
## # A tibble: 1 × 1
##       count
##       <int>
## 1 195442453

It is important to note, however, that a lot of things we want to filter out are properties of the data itself, rather than results of checks the organisation has run. For example, the following parameters all involve fields (not assertions), but are included in the general filter:

galah_call() |>
  filter(basisOfRecord != "FOSSIL_SPECIMEN", # no fossils
         contentTypes != "Environmental DNA",
         decimalLatitude != 0,   # no points defaulting to zero-zero
         decimalLongitude != 0,
         occurrenceStatus != "ABSENT", # exclude observations of missingness
         year > 1700,                   # remove very early records
         -(degreeOfEstablishment %in% c("cultivated", "captive")),  # introduced species
         -(identificationVerificationStatus %in% c("verification required", "needs_id")) # uncertain cit sci records
        ) |>
  count() |>
  collect()
## # A tibble: 1 × 1
##       count
##       <int>
## 1 171820807

User assertions

A complementary approach is to call ‘user assertions’; flags placed by users on individual suspect records. You can find records flagged in this way as follows:

galah_call() |>
  count(userAssertions) |>
  collect()
## # A tibble: 4 × 3
##   userAssertions userAssertions_label count
##   <chr>          <chr>                <int>
## 1 50005          Unconfirmed          11282
## 2 50003          Corrected             3994
## 3 50002          Verified               462
## 4 50001          Open issue              95

The categories here are a little obscure, but cross-referencing with the ALA data quality profile, it appears that only issues that are ‘Open’ (code 50001) or ‘Unconfirmed’ (code 50005) have some likelihood of being ‘true’ errors. Another point to consider is who is making these assertions:

galah_call() |>
  filter(userAssertions %in% c("50001", "50005")) |>
  count(assertionUserId) |>
  slice_head(n = 10) |>
  collect()
## Limiting to first 10 of 1,382 rows.
## Use `atlas_counts(limit = )` to return more rows.
## # A tibble: 10 × 3
##    assertionUserId assertionUserId_label count
##    <chr>           <chr>                 <int>
##  1 57159           Thomas Mesaglio        1463
##  2 124186          Cameron Slatyer        1414
##  3 2126            Niels Klazenga         1059
##  4 3334            Penelope Mills          320
##  5 8902            ralph FOSTER            235
##  6 11936           Kay Durham              218
##  7 1108            Allen Sundholm          204
##  8 29516           Wendy Potts             187
##  9 25919           Matthew Connors         177
## 10 109408          Ben Travaglini          164

This isn’t actually terribly informative unless we know who these people are. But it likely that the types of errors flagged by a single user will reflect their spatial and taxonomic knowledge of the Australian biota. To remove any records with user assertions, use is.na(userAssertions) within your filter statement:

galah_call() |>
  filter(is.na(userAssertions)) |>
  count() |>
  collect()
## # A tibble: 1 × 1
##       count
##       <int>
## 1 199050968

Concluding remarks

Data quality is an ongoing topic of investigation for researchers and data infrastructures. The tools described above are the current best available toolset implemented at the institutional level. For a more general assessment of how to handle data quality issues in R, see Cleaning Biodiversity Data in R by Dax Kellie et al. (2026, https://doi.org/10.54102/ala.77009).

A critical limitation of data quality assertions and their derivatives is that they focus exclusively on rule-based methods; there is no probabilistic component to them. Users seeking to draw probabilistic inference from these data will need to consider carefully whether there are classes of outlier relevant to their use case, but which are invisible to the systems in question.