Mastering the dbt™ CLI - Selector Methods
In this third part, we will go through selector methods in the dbt™ commands that analytics engineers can use to fine tune their model selections.
Kaustav Mitra
Aug 29, 2024
·
5
min read
In the final article in this series, we will dive into the world of selector methods for even more precision and flexibility in model selection during data transformations.
What are Selector Methods?
Selector methods in dbt™ allow you to filter resources based on specific properties using the method:value syntax giving you the power to target exactly what you need.
Wildcard Magic
Most selector methods support unix-style wildcards. Here's a quick rundown:
*: Matches any number of characters (including none)?: Matches any single character[abc]: Matches one character listed in the bracket[a-z]: Matches one character from the specified range in the bracket
Example:
The Selectors
1. Tag Selector
Use tag: to select models with a specific tag.
2. Source Selector
Use source: to select models that reference a specified source.
3. Resource Type Selector
Use resource_type: to select nodes of a specific type.
4. Path Selector
Use path: to select models/sources defined at or under a specific path.
5. File Selector
Use file: to select a model by filename:
6. FQN (Fully Qualified Name) Selector
Use fqn: to select nodes based on their fully qualified name:
7. Package Selector
Use package: to select models defined within the root project or an installed dbt™ package.
8. Config Selector
Use config: to select models that match a specified node config.
9. Test Type Selector
Use test_type: to select tests based on type.
10. Test Name Selector
Use test_name: to select tests based on the name of the test defined.
11. State Selector
Use state: to select nodes by comparing them against a previous version of the project.
This is a pretty big topic in itself with many variants. For an in-depth understanding check the Paradime docs on state selector. It’s pretty intense 😀.
12. Exposure Selector
Use exposure: to select the parent resources of an exposure.
13. Metric Selector
Use metric: to select parent resources of a metric.
14. Results Selector
Use result: to select resources based on their results status from a previous execution.
15. Source Status Selector
Use source_status: to select based on the freshness of sources.
16. Group Selector
Use group: to select models defined within a specified group.
17. Access Selector
Use access: to select models based on their access property.
18. Version Selector
Use version: to select versioned models.
19. Semantic Model Selector
Use semantic_model: to select semantic models.
20. Saved Query Selector
Use saved_query: to select saved queries.
21. Unit Test Selector
Use unit_test: to select dbt™ unit tests.
Pro Tips for Selector Mastery
Combine Selectors: Mix and match for precision targeting.
Use Graph Operators: Combine with + and @ for complex selections.
Exclude with Negation: Use --exclude to exclude certain models.
Default Method: If you omit the method, dbt™ will default to one of path, file, or fqn.
Advanced uses
Combining selector functions with graph operators, analytics engineers can create complex dbt™ commands that execute exactly the models as needed. In this section, we share with you advanced use cases that you can use in your own dbt™ projects.
1. The Kitchen Sink
What's happening here?
tag:nightly: Selects all models tagged with 'nightly'+final_model: Addsfinal_modeland all its upstream dependencies--exclude "staging.excluded_model+": Excludesexcluded_modeland all its downstream dependencies
Use case: Run nightly models and critical path, but skip a problematic staging model and its dependents.
2. The Wildcard Wizard
What's the magic here?
staging.*: All models in the staging directoryintermediate.*: All models in the intermediate directory+analytics.critical_metric: The critical metric model and its dependencies
Use case: Refresh all staging and intermediate models, ensuring a critical metric is up-to-date.
3. The Exclusion Expert
Let's decode:
tag:hourly: Excludes models tagged ashourlytag:weekly: Excludes models tagged forweeklyruns
Use case: Run everything except hourly models and weekly-only models.
4. The Intersection Inspector
What's intersecting? Selects models that are:
Tagged as 'critical'
Tagged as 'nightly'
Dependencies of
final_dashboard
Use case: Run critical nightly models that affect the final dashboard.
5. The Path Prowler
Path perfection:
path:models/staging/core: All models in the core staging pathpath:models/mart/finance+: Finance mart models and their children
Use case: Refresh core staging and propagate changes through finance models.
Wrap Up
There you have it - these advanced graph operator techniques will let you slice and dice your dbt™ project with precision. You will be able to execute exactly the models you want in your production dbt™ pipelines.
