Introduction

Today we are announcing two critical performance boosts in analytics workflow. The Paradime Code IDE, the most advanced IDE for analytics development using dbt™*:

• Scratchpad
• JIT (just-in-time) dbt™* compilation

Problem

At Paradime, we work backwards from our customers - so we want to make sure we are always solving our customers problems. When we started building the Code IDE, our hypothesis was that people would use the IDE for development and our workbench for data exploration.

As more and more of our growing customer base migrated from dbt Cloud™* to Paradime, we realized that we were wrong. We are always looking to understand and listen to our customers, and we learned our users needed two things:

1. They want to write SQL queries in unsaved, empty files in the Code IDE and run that against their warehouse. It's like how we would open a new tab in DataGrip or DBeaver and immediately start writing SQL queries.
2. They would do (1) multiple times during the model building lifecycle and expect results to be available fast.
3. They would expect the same with dbt™* models and would expect the compilation to be instant.
4. And finally, our Data Preview functionality took ages for dbt™* to compile - making it very hard to use.

Solution

Two very critical performance boosts to the analytics development experience:

Scratchpad

Now anyone can open a new tab in the Code IDE, start writing SQL and then press the Preview Data button to instantly see the results of the SQL. Scratchpad also supports dbt™* models, where we will compile and show the data preview. Users can write and query SQL in the same place alongside building dbt™* models.

JIT Compilation

The Code IDE now instantly compiles the entire dbt™* project in milliseconds. Before it used to take minutes. Under the hood, we use the insanely good dbt-osmosis package and a massive shoutout to Alexander Butler for building and maintaining this. Another great example of how open source can supercharge user experiences. Changes made to a model, like adding SQL or removing columns are reflected immediately. Opening a dbt™* model, it's already pre-compiled. This performance boost itself will make our customers day-to-day work 100x faster. Users can also see dbt™* model and compiled SQL side by side. This view can save users many hours, especially when someone is building complex models.

We ran a benchmark to see how we compared with dbt Cloud™*, and across a few hundred files, our compilation speeds were 300% faster. That's 300% more productivity for Paradime customers.

Conclusion

These performance boosts are critical to meeting the high performance demands of our growing customer base. These boosts will be rolling out to our users next week . Some of the next steps we are already thinking about are JIT linting, error detection, and many other applications to add more performance boosts to our user base.

‍

Sign up for a FREE 14-day trial or schedule some time with our team to learn more about Paradime 💪

‍