Introduction

It's already been two months since July, when we put out our last stuff we shipped, and it feels like ages ago. Over the last year or so, we have been rolling out features fast and furious, and we needed to take a pit stop to look into the finer details. The found needles in a haystack, little hard-to-find things that have a massive impact on the end user experience. We need a separate post for the 30+ small things we fixed that you used to cause user frustration. In the meantime, however, we did not stop building.

So, today I want to talk about 2 things we are announcing:

1. Product spotlight on our Code IDE search experience.
2. Paradime Live v2.0 - the next iteration of our just-in-time (JIT) dbt™ compiler - faster, resilient, and more robust than before.
3. Outgoing Webhooks from Paradime to any 3rd-party application unlocking thousands of use cases.

Search experience in Code IDE

Search is one of the most fundamental user actions in a code editor and yet lacking in many cloud-based editors for dbt™ development. With powerful search, analytics, engineers develop faster than ever before on Paradime compared to other cloud IDEs like dbt Cloud™.

In Paradime, analysts can:

1. Search in a file using ⌘ / Ctrl + F
2. Search across files using case-sensitive, insensitive, and regular expressions and select which files should be included or excluded while you search
3. Replace and replace all when you search
4. Search for files across your repo

Using the Paradime search experience, you can:

• Open a file by searching instead of looking across your file tree in dbt Cloud™
• Search and replace a column or table or model name across your entire repo in 2 clicks
• Powerful selectors to search and replace in exactly where you want

These are table-stake ergonomics that developers need to be productive in their day to day analytics development. We see time and again how delighted our users get when they see themselves unblocked on Paradime Code IDE.

Paradime Live v2.0

In May, 2023 we shipped the first iteration of our just-in-time dbt™ compiler, Paradime Live, built on top of the awesome dbt-osmosis project.

The initial idea for Paradime Live was to:

1. Allow users to write SQL in scratch files, compile, and run them on the fly so they can see data behind the code in real-time
2. This would also apply to existing dbt™ models that were being developed and modified as part of the normal development process.

As our user base grew, we learned that there are a lot more use cases under which Paradime Live needs to work like:

• Adding / updating warehouse connections
• Changes and updates to dbt_project.yml
• Updating environment variables
• Updating dbt ™versions
• Handle model compilation and previewing data separately
• Compile model during data preview so its based on the freshest code

There was another associated issue with dbt-osmosis that it ran in the same memory space as the rest of the IDE. For large dbt™ projects, dbt-osmosis would then run into an OOM or out-of-memory issue that would crash the IDE for the user. This is because the underlying infrastructure would simply run out of memory.

So we needed something that was fast, memory-isolated, and could handle various use-cases that our users needed. And there comes Paradime Live 2.0.

With Paradime Live 2.0, users can choose to just compile their code or compile and run it at the same time.

When users change or update environment variables, dbt_project.yml and other settings that have a global impact, we re-compile the entire project so that fresh compiled files are available quickly.

We do the compilation in a separate process with its own cpu and memory allocation to handle projects of any size and complexity.

We have refreshed the UI too, so that now users can see exactly what is going on with informative zero states.

This is a ground-breaking architecture change and an incredible achievement by the entire team to pull this off in a matter of weeks.

Outgoing Webhooks

Webhooks are a way for an app to provide other applications with real-time information. When an event happens in one application, a webhook can notify another application that is set up to receive notifications. This allows applications to communicate and share data in real time.

Outgoing webhooks are now live for all customers in the Growing Tier onwards. We are starting with 2 events as follows:

• bolt.run.started - triggered when a Bolt run is started
• bolt.run.completed - triggered when a Bolt run is completed

How does it work?

Customers just need to add a webhook endpoint, and subscribe to the events they want us to post to that endpoint, and that's it.

Depending on use-case, users can enable rate-limiting and custom headers, too.

In the event catalog tab, we show all the details of the events and what the response schema looks like.

In the logs and activity tab, all audit and activity logs are available out of the box so that users can jump of message and troubleshoot.

Use cases

Using webhooks, we are already seeing our customers unlock new use cases previously not possible, for example:

• Get a notification for a failed dbt™ run in Slack as a message and for every additional failure of the same schedule, get notified as threads
• Get notified of run statuses in Microsoft Teams
• Create incidents in PagerDuty, OpsGenie, Splunk, incident.io, FireHydrant or other tools if a schedule fails
• Create custom notifications in Slack or Teams depending on schedule name, schedule severity
• When schedule fails, automatically create tickets in Jira, Linear, or any other tool and assign the ticket to the schedule owner
• Hook up to Zapier or make.com and connect to 1000+ applications and unlock infinite possibilities.

Conclusion

This is it for now, but we have some pretty cool stuff coming up for analytics teams in the next few days. For folks heading to Coalesce, Fabio and Parker will be there so please drop by to say hi or to know more about Paradime.

‍