The October release of data.world brings a wide variety of new capabilities and improvements across the platform – read on to learn more about the GA of Databricks Publisher, new collectors for MongoDB and Alteryx, Okta support in SCIM, the GA of the improved search experience, and more!

Additionally, we highlight some changes made to the data.world Open Data Community to improve privacy and preserve the quality of open data and the user experience.

Databricks Publisher Premium Automation

We’re excited to announce the GA launch of the Databricks Publisher Premium Automation! This new feature allows users to seamlessly publish metadata from data.world to Databricks, simplifying the process of managing and synchronizing key data attributes. Specifically, users can now automatically publish table and column descriptions from data.world to Databricks and push selected metadata attributes as Databricks tags. Whether you prefer manual updates or fully automated syncing, this automation ensures that metadata remains consistent between platforms, reducing manual effort and improving data integrity. With data.world now acting as the source of truth, your metadata stays up-to-date across systems effortlessly.

For more information, see the product documentation.

New MongoDB and Alteryx Collectors

This month, we’re excited to announce new MongoDB and Alteryx Collectors, both available in Private Preview. If you’re interested in early access to either of these new collectors, please reach out to your Customer Service Director.

MongoDB Collector

The MongoDB Collector catalogs metadata from MongoDB, helping maintain a comprehensive inventory of MongoDB assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for MongoDB databases, collections, views, indexes and more.

An example collection from MongoDB

Alteryx Collector

The Alteryx Collector catalogs metadata from Alteryx, helping maintain a comprehensive inventory of Alteryx assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for workflows, workflow nodes, workflow jobs, connections, schedules and more.

An example collection from Alteryx

Improved lineage for SQL Server

The SQL Server Collector now collects additional lineage relationships not previously captured through SQL parsing using built-in SQL Server functions that describe relationships between objects (such as, in some cases, the columns and tables referenced by views or stored procedures).

For more information and detail, see the description of lineage collected by the SQL Server Collector in the product documentation.

Support for Okta in SCIM

The active Private Preview of SCIM (System for Cross-domain Identity Management) now additionally supports Okta (in addition to Microsoft Entra ID), allowing customers who use Okta as their enterprise identity provider to have automated management of users and groups in data.world.

If you are interested in being part of the SCIM Private Preview, or just want to learn more, please reach out to your Customer Service Director.

Webhook Authorization enhancement

Webhooks now support an optional authorization key parameter to help consuming applications verify the origin and permissions for an incoming webhook. Learn more

Collection Details in Technical Reference

By popular request, the Technical Reference page for catalog resources now includes details about the collections the resource belongs to. Learn more

Relative Time Advanced Search Syntax

Create powerful saved searches for resources by updated and created dates using three new relative time options:

• `created:today` 
• `updated:yesterday`
• `created:{last 30 days}`

See the product documentation on creating advanced searches to learn more.

UX Improvements

Adding the new search experience to Organizations: Our new search experience has been a big hit with users. It’s faster, cleaner, and provides more advanced features in the UI. We've fully retired the classic experience and brought the new search features to the Resources, Glossary and Collection landing pages.

Coming Soon! Advanced relationship editing: We're adding new improvements that make it easier to find the right resources and add or remove more than one relationship at a time.

Coming Soon! More look-and-feel updates: Next up in our work to update and modernize our UI, we'll be swapping the old default avatars to a newer color palette and default avatar design that utilizes letters. This change will also provide a more accessible experience as it gives users the ability to distinguish users and organizations using letters.

Changes to data.world Open Data Community

data.world Open Data Community profiles, datasets, and projects now behind a login wall: To better control the privacy of our users and to protect the effectiveness of the content on our active Open Data Community, we have made the decision to restrict access to profiles, datasets, and projects to account holders. It is always free to join our open data community. 

data.world Open Data Community commenting restrictions: Commenting is now restricted to contributors on datasets and projects in the data.world Open Data Community. Organizations can enable comments on their public datasets through organization settings. This feature is not available or enforced for Enterprise customers on private instance or VPC deployments.

The September release of data.world is live, and is jam-packed with features and improvements that will make catalog teams and users more effective! We’ve got a major improvement to the Search UX, support for SCIM to improve access management, and new collectors to continue making your data.world catalog the center of your organization’s data ecosystem.

Read on to learn about these and other exciting new features!

Simplified Search Experience

We’re excited to announce that our new and improved search experience is now generally available! Our new experience has been in public preview since mid July. Through the preview feedback as well as months of user research, we’ve made significant updates to ensure a more intuitive and streamlined search experience.

With a cleaner interface and new features, users can now find what they need faster and with fewer clicks. Enjoy faster discovery through quick organization scoping; build, save, and share custom filters; pin and order your own top facets; preview the resource without leaving search; view important context like resource hierarchy and custom metadata; and utilize inline advanced features without leaving your current search.

Why the change? We listened to your input and designed a new search experience that eliminates noise and complexity, making it easier for all users—from new users to experienced catalog pros to enjoy faster, more efficient searching. 

You can learn more about these improvements in our product documentation and our Searching and Exploring Data course in data.world University.

Notable UI and UX improvements

Relating Resources: Resource relationships are important. We’ve made small but meaningful improvements to the user experience for relating resources to one another. Users will now have more room and more context as they search and relate resources. 

Providing Feedback: We want to hear directly from end-users so that we can provide an even better product. We’ve built a simple in-app feedback form that allows us to get direct feedback from users.

Look and Feel: We’ve updated our typography, modernized actions and components, and provided more space to absorb information. This is the first in a series of user interface improvements we’ll be rolling out over the coming months.

SCIM - Microsoft Entra (Private Preview)

Imagine never worrying about managing user access again! We are excited to introduce support for SCIM in data.world, to streamline the process of managing catalog, organization and resource access and permissions.

SCIM (System for Cross-domain Identity Management) makes user management automatic, ensuring the users in your enterprise always have the right access at the right time, by syncing user details and group membership between your identity provider and data.world. From onboarding to offboarding, every detail – job title, email, and permissions – stays perfectly synced. Why does this matter? Because separate data entry and manual errors can lead to security risks, manual effort, and frustration. With SCIM, you’re not just saving time; you're enhancing security and efficiency, protecting your business, and making sure nothing slips through the cracks. It’s peace of mind, effortlessly.

This Private Preview release of SCIM functionality supports Microsoft Entra (a.k.a. Azure Active Directory) as an identity provider. We will be adding Okta support in the near future.

If you’re interested in learning more about SCIM support in data.world, or accessing the Private Preview, reach out to your Customer Service Director.

Qlik Sense Cloud Collector (Private Preview)

The Qlik Sense Cloud Collector catalogs metadata from Qlik Sense Cloud, helping maintain a comprehensive inventory of Qlik Sense Cloud assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for apps, visualizations, sheets, fields, measures and more. For more information, see the product documentation.

An example collection from Qlik Sense

Informatica Cloud Data Integration (CDI) Collector (Private Preview)

The Informatica Cloud Data Integration (CDI) Collector catalogs metadata from CDI, helping maintain a comprehensive inventory of CDI assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for jobs, mappings, mapping tasks, and more. For more information, see the product documentation.

An example collection from Informatica Cloud Data Integration

Amazon QuickSight Collector (GA)

The Amazon QuickSight Collector, previously in Private Preview, is now generally available! The QuickSight Collector catalogs metadata from QuickSight, helping maintain a comprehensive inventory of QuickSight assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for Analyses, Dashboards, Datasets, Data Sources, and Folders. Additionally, lineage metadata is harvested between QuickSight Analyses, Datasets, and Data Sources when the data source is a relational database, S3, or Athena. For more information, see the product documentation.

An example of a QuickSight Dashboard

Search API + Syntax improvements

Public search API endpoints and all UI search interfaces now support advanced syntax to target specific metadata fields when constructing complex searches. The following keywords and partial match syntax is now supported:

title:”Sales Order"
Searches for exact matches on the title field for the value “Sales Order”

title:”*sales”
Searches for matches with titles containing the term “sales”.
The leading '*' character is used to specify partial or fuzzy match.

“description” and  “summary” can also be used as keywords to target specific fields.

Custom metadata can be searched in this manner with the following syntax:

metadata:”Submitted by:Tim Gasper”
Searches for exact matches on the “Submitted by” field with the value “Tim Gasper”

metadata:”Steward:*Juan”
Searches for matches on the “Steward” field that contain the term “Juan”

When using the public search APIs, specific fields can be targeted for exact and partial matching using the property object and the desired field IRI:

{
    "owner": "democorp",
    "property": {
        "https://democorp.linked.data.world/d/ddw-catalogs/steward": "*Gasper"
    }
}

The August release of data.world brings a number of new and improved product capabilities, including an improved user interface for resource creation, real-time metadata sync with Databricks, a new metadata field to improve an understanding of where catalog resources come from, and enhancements to Microsoft, Salesforce and Databricks collectors.

Also available now is an exciting improvement to our AI Context Engine™ that helps provide explainable answers from your structured data.

Read on to learn about these exciting new features!

Active Directory authentication for Microsoft Collectors

The SQL Server, SQL Server Reporting Services, Power BI Report Server, and SQL Server Integration Services Collectors now support Active Directory domain credentials using NTLM authentication type allowing the collector to connect securely using Active Directory-managed authentication.

Salesforce Collector

The all-new Salesforce Collector catalogs rich metadata from Salesforce, helping maintain a comprehensive inventory of Salesforce assets, facilitating better governance, discovery, and utilization of data across your organization.

The new version of the collector now harvests metadata for objects, fields, dashboards, and reports directly via Salesforce APIs.

An example collection from Salesforce

Databricks Collector harvests lineage to Amazon S3 and ADLS Gen2

The Databricks Collector now harvests External Locations allowing users to understand cross-system lineage between Databricks, Amazon S3, and Azure Data Lake Storage Gen2.

AI Context Engine - new "detailed answer" endpoint

The new "detailed answer" endpoint in AI Context Engine works similarly to the existing "Answer Tool" endpoint, but it returns much more information, including:

• answer - Textual response, same as before
• result - raw data & schema (frictionless data format)
• sparql - SPARQL query
• sql - the SQL query
• targetSql - SQL queries executed against target systems
• terms - Business terms that were used to generate the query
• ontologyUsed - The parts of the ontology that were used to generate this response
• evidence - the "thoughts" that were generated during the run (same as what you might see in the debug chat tool, Archimedes)

In contrast, "tool" endpoints are simpler, returning only the response in order to integrate seamlessly with other LLMs (e.g., OpenAI).

In the future, this and other "detailed answer" endpoints in AI Context Engine will return additional information and evidence as we further deliver on accurate, explainable and governed answers from your structured data.

Link: https://developer.data.world/reference/callanswer

Source System metadata field

Source System is a new default field that consistently describes the system from which the catalog record metadata was sourced (e.g. Tableau). This field will be used to improve discovery by allowing types to be organized by Source as well as helping to differentiate between ambiguous resource type names (e.g. Dataset). Read more about how to extend and configure this field for your custom types and collectors here.

For more information see our product documentation.

Databricks Publisher real-time updates

This feature allows automatic triggering of Databricks Publisher automation whenever a Databricks Column or Table description is updated in the data.world catalog via the UI or public API. This ensures real-time synchronization of metadata between data.world and Databricks, eliminating the need for manual updates.

Databricks Publisher (announced last month) is currently in Beta, so to get access, reach out to your Customer Success Manager.

Improved UX for resource creation

We’ve enhanced the UI for creating new resources in data.world. Now, when users create a resource in data.world, a new multi-step wizard flow replaces the old small pop up modal.

This new approach makes resource creation much easier for a wider variety of users - thanks for your feedback on this important aspect of the catalog user experience!

The July release of data.world is here, with starter kits to get AI Context Engine™ up and running quickly, enhancements to collect more metadata for popular collectors, the first release of Databricks Publisher to keep metadata in sync between data.world and Databricks, and an opt-in preview of an improved search experience within data.world.

Read on to learn about these exciting new capabilities, available now!

AI Context Engine™ Starter Kits

To help customers quickly utilize the AI Context Engine, we now offer three starter kits:

• Slack
• Microsoft Teams
• Streamlit

Key Benefits

• Quick Start: Each starter kit provides all necessary components to get up and running quickly.
• Customization: Clients can modify the source code to suit their specific needs.
• Ongoing Updates: Access the latest features and improvements by updating from GitHub.
• No Custom LLM Required: The starter kits enable direct AICE API calls for basic Q&A on structured data.

Availability and Support

The starter kits are available to all AICE customers at the links above. While they are provided as-is and unsupported, they offer a robust foundation for developing custom applications.

Enhancements to Power BI, dbt, and Denodo Collectors

This month, we’re excited to announce enhancements to our top collectors to harvest more metadata and lineage relationships.

Power BI Collectors enhancements

The Power BI Service and Power BI Gov Collectors have been updated to harvest preview images for Power BI reports allowing users to preview reports in data.world before they navigate to them in Power BI.

dbt Collector enhancements

The dbt Core Collector has been updated to streamline the setup of multiple dbt Core Collector instances allowing users to specify multiple run_results.json files in a single run. Additionally, the dbt Core and dbt Cloud Collectors now support Azure Synapse as a target database.

Denodo Collector enhancements

The Denodo Collector has been updated to harvest lineage between Denodo resources and cross-system lineage between Denodo and Power BI are now supported.

Announcing the Beta Release of Databricks Publisher!

We are excited to introduce the first version of the Databricks Publisher, launching in Beta on July 23. This new feature allows you to write back Databricks column and table metadata for individual resources with just the push of a button, streamlining your metadata governance processes.

What’s New?

The Databricks Publisher enables seamless synchronization of metadata, ensuring that annotations by subject matter experts and data stewards in data.world are reflected back in Databricks. This enhancement benefits analysts and data scientists by providing well-governed, meaningful metadata directly within their Databricks environment.

But that’s not all! We have more exciting features in the pipeline:

• Real-Time Updates: Soon, you won’t need to manually sync changes. Any saved changes will automatically update in Databricks.
• Tag Writeback: This upcoming feature will extend the functionality to include tags, further enhancing your governance capabilities.

Why This Matters

Synchronizing metadata between data.world and Databricks creates a seamless governance workflow. This integration supports a broader persona model where end users and governance professionals utilize data.world directly, while technical analysts and engineers use platforms like Databricks for discovery and data work. By bridging these environments, we’re making it easier for your teams to collaborate and access the data they need.

To get access to this Beta feature, reach out to your Customer Success Manager. We can’t wait for you to experience the benefits of this new feature and look forward to your feedback as we continue to enhance our offerings.

Stay tuned for more updates, and thank you for being a valued customer!

Public preview of new search experience now available

We're happy to announce the public preview of our new search experience! As part of our continuous effort to improve your experience with our platform, we've made some significant upgrades and introductions to the way you search data on our product. For the next month, we invite you to opt-in to the new preview where you can interact with these new features before they go live.

Our enhanced search function now offers a preview of key resource elements, allowing you to quickly differentiate results and jump to related resources or explore resource lineage without steering away from your search experience. Some much-loved additions include refined and streamlined filters to help you find exactly what you need, more efficiently. You can now enjoy a search for filters, select more than one, and delve into advanced filter operations (ANY, ALL, NONE). Organization scoping has never been smoother – stay within your org scope, and sort regardless of your scope.

We’ve made the experience cleaner and more user-friendly, cutting down on overwhelming filter options and obscure filter values, and enabling a customizable order of filters – pin your favorite or most-used ones to the top for ease. Plus, you can also build and save your own filters for future and repeated use.

These changes root from the invaluable feedback we received from you,  our valued customers and users. You asked for a simpler, more manageable experience, the ability to preview more types, as well as personal customizations like saved searches and re-ordered facets – and we listened! We’re incredibly excited for you to experience these advancements. You can read more about what we’ve changed in our documentation portal.

The interface invites you to share your feedback directly within the app, or you can reach out to your Customer Success Manager to voice your thoughts. We look forward to hearing from you!

The June release of data.world is here – featuring a new collector for Microsoft SSIS, enhancements to collect more metadata for Databricks and PowerBI, the introduction of versioning for Governance Automations, and a useful set of new security and management features for catalog admins and data stewards.

Read on to learn about these exciting new capabilities, available now!

Enhancements to Databricks and PowerBI Collectors, and a new SQL Server Integration Services Collector.

This month, we’re excited to announce improvements to some of our most frequently used collectors, and the new SQL Server Integration Services collector.

The new SQL Server Integration Services Collector is available in Private Preview, contact your Customer Success Director to learn more how to participate in the program.

SQL Server Integration Services Collector in Private Preview

The SQL Server Integration Services (SSIS) Collector catalogs metadata from SSIS, helping maintain a comprehensive inventory of SSIS assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for projects, packages, control flow/data flow executables, and much more.

An example collection from SSIS

Power BI Service Collector enhancements

The Power BI Service Collector has been updated to harvest Power BI Measures (including lineage) and lineage between calculated columns. Additionally, the collector now harvests lineage from Power BI Query statements and supports lineage between upstream data sources configured using ODBC connections.

Databricks Collector performance enhancements and external locations

A number of performance improvements were made to the Databricks Collector when harvesting tags on Databricks tables and lineage metadata from Unity Catalog. Users may see up to 80% improvement depending on the shape/weight of their Databricks instance. The Databricks Collector has also been updated to harvest external locations (Azure Data Lake Storage Gen2 and Amazon S3).

Start exploring today

These new collector updates help users understand where data in these reports are sourced from, facilitating troubleshooting for analysts and increasing trust for business end users. Learn more in our documentation:

• Snowflake Collector documentation
• SQL Server Integration Services Collector documentation
• Power BI Service Collector documentation
• Databricks Collector documentation

Versioning for Governance Automations

This new functionality allows admins of Governance Automations in data.world to:

• Edit Existing Automations: Modify your current automations directly within the system, ensuring they meet your evolving needs.
• Maintain Task Integrity: Existing tasks, both claimed and unclaimed, stay connected and unaffected by changes.
• Future-Proof Configurations: Any new runs initiated post-edit will use the updated configuration.
• Stay Updated: Edited automations automatically update to the latest template version, incorporating the newest features and capabilities.

Enjoy the additional flexibility and power of Editable Automations, available today!

Data Exfiltration Controls

Enterprise customers can now configure instance-wide policies for downloading dataset content from the platform. This suite of controls also adds support for restricting who can create personal access tokens for the data.world public API. 

Learn more about this feature by visiting our documentation portal.

Organization Browse Card Wizard

Organization admins can now design, build, link resources, and edit the organization level browse card through a visual wizard in the UI. This feature is compatible with more advanced Browse Card configuration options such as automations.

Learn more about this new feature by visiting the documentation portal.

Administrative Functions for Discussion Topics 

Resource admins now have the ability to manage Discussion Topics on the Discussions tab of a resource to help maintain highly useful and accurate content. An admin can delete discussion topics as well as edit the title of an existing topic.

Learn more about this new feature by visiting the documentation portal.

The May release of data.world has something for everyone – user experience improvements for navigation and understanding, a new source of technical lineage, big performance improvements across multiple collectors, and a set of powerful and time-saving capabilities for the admins and program teams building and managing their catalog experience.

Read on to learn about these exciting new features!

New relationships summary and browsing

In our continued effort to streamline the user experience, we've rolled out a few changes to our catalog metadata resource details pages. These adjustments have been carefully designed to save valuable time and provide the information needed to understand and navigate related resources. Stay tuned for more updates to these pages in the coming months!

Collector performance improvements and Oracle lineage

This month, we’re excited to announce improvements to the overall runtime for a number of our collectors and an update to the Oracle collector for harvesting lineage metadata.

We are also excited to announce that both the Amazon DynamoDB Collector and Azure Data Factory Collector are now generally available.

Performance improvements across collectors

A number of performance improvements were made to collectors to improve their overall runtime and reduce memory usage. Customers may see up to 90% improvement depending on the collector and shape/weight of their database/data warehouse.

These collectors include Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server, dbt Core, and dbt Cloud collectors.

Oracle collector harvests lineage metadata

The Oracle collector now harvests lineage relationships from Oracle views, stored procedures, and functions. With this new metadata, users can now visualize and query for how data is moved within Oracle and other technologies.

Start exploring today

These new collector updates help users catalog their sources faster, facilitate troubleshooting for analysts, and increase trust for business end users. Learn more about what is supported in our documentation:

Oracle Collector documentation

Organization Details Public API

We’ve added a utility endpoint to our public API to surface organization details such as extended description and avatar for use in integration development. Visit the developer portal to learn more.

Catalog Resource Public API

We're delighted to announce an updated suite of API endpoints focused on flexible catalog management. 

We’ve seen wide adoption of advanced catalog features like custom resources, relationships and integrations. Our public API now has full support for our “catalog anything” mission and brings the flexibility of the knowledge graph to the initiatives you are working on, both on and off the data.world platform.

Visit our interactive developer portal to learn more and try out the new functionality.

In-App Technical Reference

As a companion to the Catalog Management APIs, we’ve added a new in-app reference to each resource page that provides in-depth information about the ontology and configuration details for the resource. The Technical Reference page can be found by navigating to the “Settings” tab of any resource and clicking “Technical Reference” in the left navigation menu.

Use this reference as a starting point for taking full advantage of the power of the knowledge graph through SPARQL queries and our Public API.

The reference provides details about the supported relationships, metadata fields, selection values, asset statuses and type inheritance for the resource.

For a comprehensive introduction to the Technical Reference, visit our documentation portal.

User Management Utilities

Administrators with the Instance Admin role now have expanded capabilities for managing active users of the platform. Administrators can now self-serve on deactivating users when someone leaves the company or should no longer have access to the system. 

We’ve also enabled instance admins to promote other users to the role without needing to do so through data.world support. 

Access Audit Utility

Administrators are often asked to troubleshoot access issues and confirm that access has been appropriately revoked when users change roles or need help. The user management portal includes a quick reference utility to check the access level a user has to any resource on the platform.

You can learn more about both of these features in our documentation.

April brings multiple new metadata collection capabilities to data.world, including Collector enhancements for Snowflake and Databricks, and a new Collector for Amazon Managed Streaming for Kafka.

Read on to learn about these exciting new features!

Catalog Snowflake Streamlit Apps, Databricks Tags, and Amazon Managed Streaming for Kafka (MSK) assets

We’re excited to announce updates to the Snowflake and Databricks collectors to harvest more metadata and collector support for Amazon Managed Stream for Kafka. These updates gather more metadata from these systems and seamlessly bring it into our data.world platform. This metadata helps both technical and non-technical users discover and understand their data quickly, govern their data with greater context, and increase trust in data by providing information about data health and transformations.

All new features are generally available.

The Snowflake Collector harvests metadata from Streamlit in Snowflake

The Snowflake Collector now catalogs metadata from Streamlit in Snowflake, facilitating better governance, discovery, and utilization of Streamlit apps across your organization.

The metadata harvested for Streamlit apps includes comments, owners, creation date, and root location. From data.world, users can discover apps and navigate directly to the app in Snowflake.

An example of a Streamlit app

Databricks Collector harvests Databricks tags

The Databricks Collector now catalogs tags from Databricks catalogs, schemas, tables, and columns. Tags are used in Databricks to simplify the search and discovery of data assets. With these tags now in data.world, users can quickly discover data assets in Databricks. For instance, product teams can now build their data products in Databricks and identify them in data.world.

An example of Databricks Tags

Amazon Managed Streaming for Kafka (MSK) Collector

The new Amazon Managed Streaming for Kafka (MSK) Collector catalogs metadata from Amazon MSK, helping maintain a comprehensive inventory of MSK assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for clusters, brokers, topics, consumers, and producers.

An example collection from Amazon MSK

Start exploring today

These new collector updates help users understand where data in these reports are sourced from, facilitating troubleshooting for analysts and increasing trust for business end users. Learn more in our product documentation:

• Snowflake Collector documentation
• Databricks Collector documentation
• MSK Collector documentation

March brings a host of new capabilities to data.world, including a new Snowflake integration for Tag Syncing, two new collectors (Power BI Report Server, Amazon QuickSight), a highly-requested interface improvement to better understand relationships, and a Chrome Extension for Hoots.

Read on to learn about these exciting new features!

Snowflake Tag Sync Automation [beta]

This automation allows users to edit and create new Snowflake tags within the data.world platform and then sync those Snowflake tags between Snowflake and data.world.

Key Features:

• Easily edit and create new Snowflake tags using data.world’s simple user interface
• Sync edited/new Snowflake tags back to Snowflake with the push of a button
• Display Snowflake tags in a new section titled "Snowflake Tags" on resource pages
• data.world becomes the source of truth for Snowflake tags when this automation is enabled

Why integrate your Snowflake tags in data.world?

Creating and editing tags is a breeze within data.world’s UI. Snowflake tags are powerful governance tools that allow users to apply policies, control access, and discover resources. An easier method for users to create and edit tags via the data.world means it’s easier to govern Snowflake resources.

Inside the data.world platform, you can view Snowflake Tags on Snowflake resource pages, like this Column page. You can also Sync the tags back to Snowflake with a simple push of a button.

The Snowflake Tag Sync Automation is currently in beta and is available as part of the Data Governance Premium offering. If you are interested in this feature, please reach out to your Customer Success Director and they will help enable the feature for you. You can read our product documentation here for full details.

Relationships as Fields

You asked, we listened. Our latest improvement is crafted from the desire to streamline the enrichment experience, making it easier to build, manage, and see important relationships. This capability allows metadata fields to be built using custom relationships between resource types, providing a more intelligent way to manage metadata and inspire users to build relationships. For example, if you cataloged your Teams and Data Products, you might want to create a relationship to show which teams govern which products (screenshot below). Your users might see and navigate to Team resources via Data Products, or vice-versa.

These new types of fields help users see, understand, and navigate relationships and show the knowledge graph at work. This enhancement compliments our flexible architecture that allows you to build custom Types, Fields and Relationships - deciding what to catalog and how best your users might want to navigate the related resources. This feature focuses on the philosophy that there are many kinds of relationships - some of which have an "attribute-like" utility, rather than just a related object. 

You can read more about how you might use this feature in our documentation. This is now available to all enterprise customers using either MDP or Catalog Toolkit for configuration. 

We hope you enjoy the new opportunities this enhancement brings to your catalog.

New collectors for Power BI Report Server and Amazon QuickSight

We’re excited to announce two new collectors: Power BI Report Server Collector and Amazon QuickSight Collector, which gather metadata from these systems and seamlessly bring it into the data.world platform. This metadata helps both technical and non-technical users discover and understand their data quickly, govern their data with greater context, and increase trust in data by providing information about data health and transformations.

Both new collectors are available in Private Preview, please contact your Customer Success Director if you are interested in participating in a Private Preview program. More information is available in our product documentation: Power BI Report Server collector, Amazon QuickSight collector.

Hoots Browser Extension for Google Chrome

Now available in the Chrome App Store is the data.world extension for Google Chrome, with the exciting new capability of automatic display of the Hoots badges on the data products where your organization’s users are using data and making decisions.

Now the valuable data trust signals, related glossary terms and additional context from your data.world catalog Hoots configuration are more easily displayed on your BI & analytics applications. With the Hoots Browser Extension capability, Hoots can be shown on Tableau, Power BI, Looker, and any other web-based application, and there’s no integration required to embed the Hoot display.

More information about the Hoots Browser Extension is available in our product documentation: Using Google Chrome Extension for Hoots.

We are excited to announce the launch of new features and latest improvements:

• Cloud Collectors - configure and run collectors hosted by data.world NEW
• Support for Snowflake Data Quality - collect and catalog Snowflake Data Metric Functions (DMFs) NEW
• Bulk operations UX improvements - streamlined bulk enrichment workflow  IMPROVED
• Enrichment and discovery UX improvements - more context and default sorting  IMPROVED

Read the sections below for full details on each new feature!

NEW Introducing: Cloud Collectors!

We are excited to announce the launch of Cloud Collectors, the newest way to collect metadata on data.world!

Now, you can configure and run collectors that are hosted by data.world with just a few clicks! This feature not only provides a no-code way to start bringing metadata into your catalogs faster, it also has robust functionality around scheduling and monitoring to make setup more transparent and seamless. If you have cloud-accessible data sources that you're ready to bring into your catalog, this feature is for you!

👩‍💼 How can I use Cloud Collectors?

Users with Admin access will see a new option in the collector setup wizard that says "Cloud."

Once you enter your source information, you will be able to set a custom name for your collector configuration, and set a schedule for how frequently the collector should run.

After a collector completes, you will see the metadata and resource types that were collected, as well as the source information you entered while setting up the collector. Here you will also find what might have gone wrong if the collector run failed, and you'll have the ability to cancel the run as well.

You can view all of the collectors you have set up, whether they are from collectors that you host or Cloud Collectors, on the Metadata Collection tab. From here, you can view, edit, and delete collector configurations. And if you're setting up multiple collectors for one source with the same credentials, try the "Duplicate Configuration" button to quickly set all of them up.

For a full list of supported sources and more details on the feature, please refer to the documentation here.

NEW Announcing support for Snowflake Data Quality

We are thrilled to introduce an exciting addition to our existing Snowflake collector – support for Snowflake’s brand-new Data Quality feature, currently available in private preview. This enhancement empowers users to elevate their data quality assessment to new levels.

Key Features:

📊 Collect and catalog Snowflake Data Metric Functions (DMFs): Users can now measure the quality of their data using Snowflake’s powerful "data metric functions" (DMFs) and catalog this context with data.world. Example DMFs include Null Count, Unique Count, and Freshness – providing comprehensive insights into the health of your data.

🔍 Find and understand data quality metrics: The DMFs and observations (recorded metrics) are seamlessly integrated into resource pages on your data.world platform and are also presented as Hoots associated with Snowflake tables and views. This user-friendly interface makes it easy for individuals across your organization to discover and understand data quality metrics effortlessly.

Why Snowflake Data Quality & data.world?

🌐 Compliance & Consistency: In today's data-driven landscape, ensuring compliance and consistency is paramount. This Data Quality feature & integration help you meet these standards by offering real-time insights into critical data metrics.

🔒 Build Trust: Trust is the foundation of effective data utilization. This Data Quality integration  helps users to trust their data by bringing metrics related to freshness, blank values, and inaccuracies to the catalog and everyday tools, such as Tableau and Power BI, via Hoots.

Who Benefits?

👩‍💼 Data Stewards, Engineers, Admins: Empower your data stewards and technical teams by providing them with a tool that gives immediate insights into the current state of their data based on specific metrics.

🚨 Data Consumers: With Hoots, you can identify and take swift action on tables and views that require attention, ensuring data quality monitoring is seamlessly integrated with considerations for cost, consistency, and performance.

Experience a new era of data quality and reliability with data.world’s support for Snowflake's Data Quality today!

Note: Snowflake Data Quality is an enhancement within the existing Snowflake collector, and is currently available to Snowflake Private Preview customers. ❄️🚀

Using Hoots, users can quickly see data quality issues, like duplicate data, and easily fix the errors.

Improvements and Enhancements

IMPROVED Improvements to Bulk Operations UX

Bulk operations are a crucial part of keeping a catalog updated and accurate. We're excited to announce some improvements that will streamline and accelerate bulk operations such as bulk editing tags and attributes and bulk moving resources between collections.

First, we have consolidated these operations into a single menu for each place you can initiate a bulk operation (the Glossary tab, the Resources tab, and the Collection Contains tab). Now you can Quick edit, Add resources to collections, and Export/Import resources from all three locations.

Next, we've added the granular selection experience, that previously existed only in Quick edit, to the Export/Import spreadsheet flow as well. This is available on all three entry points (Glossary, Resources, Collections), which should significantly reduce the time it takes to make changes via the spreadsheet option.

Finally, we've simplified and clarified the experience around moving resources between collections. Previously this experience only existed within the Quick edit flow, but now you can select 'Add to Collections' or 'Move or Add Collections' to access this functionality. From the Glossary and Resources tab, you'll be able to add resources to one or multiple collections, and from the Collection tab (example below), you'll be able to add resources to one or multiple collections, or move resources from one or all collections to one or multiple collections.

With these improvements, administrators and curators will be able to perform bulk operations on resources much more quickly. For more information, please refer to the documentation for bulk editing resources here and for bulk editing glossary here.

IMPROVED Added context in various search experiences

The suggested search dropdown  now has more context, including the list of collections, owning Organization or User profile, and more. We’ve also added more context to the search experience when a user is relating one resource to another. This added context makes it easier to see and understand what has already been added.

IMPROVED Default sorting improvement + column index sort

We’ve provided a default sort experience that makes scanning the related, contained, and column resources faster. We also added column index as a sort option so users can understand the original column order from the database.

IMPROVED Expansion of the Summary field

The Summary field is now available on all resource types out-of-the-box. The field is available on all Types without the need for configuration.

IMPROVED Rich Text Editing without Markdown

Multi-line fields on catalog resources support Rich Text for more engaging and understandable content, and now these fields can be edited in a What-You-See-Is-What-You-Get (WYSIWYG) user experience rather than users having to create and edit content using Markdown.

Markdown editing is still available for users that prefer it, but now more data owners and users can create compelling rich text content.

We’re excited to announce three new collectors: Azure Data Factory (ADF), DynamoDB, and Teradata. These collectors gather metadata from these systems and seamlessly bring it into our data.world platform. This metadata helps both technical and non-technical users with discovering and understanding their data quickly, governing their data with greater context, and increasing trust in data by providing information about data health and transformations. 

Azure Data Factory Collector: Detailed Data Tracking

The ADF Collector allows users to understand how your data was moved or transformed, the format changes it underwent, and its migration journey to build a foundation of trust. This collector fetches metadata for Factory, Pipeline, Activity, Linked Service, Dataset, Dataflow, Trigger, Integration Runtime, and Global Parameter within Azure Data Factory. It also provides column-level Lineage, showing how data moves between ADF Datasets and connected sources like Snowflake, Databricks, S3, and ADLS. This helps users understand data movements and transformations, increasing trust. It also allows monitoring of pipelines for health checks, boosting confidence in data integrity and reliability.

DynamoDB Collector: Simplified Discovery

Our DynamoDB Collector helps users discover and understand DynamoDB resources. This collector captures deep metadata for Tables and Streams. It’s useful for both technical users managing DynamoDB resources and non-technical users exploring metadata and understanding how they can use DynamoDB resources through an intuitive interface. Technical users will appreciate getting insight into DynamoDB resources, including Tables, Keys, Indexes, and more. 

Teradata Collector: Comprehensive Data Insight

The Teradata Collector allows users to see a holistic view of all their Teradata assets to help them manage and discover their data. This collector covers metadata for Database, Table, SQL Procedures, User Defined Functions, View, External Procedures, Triggers, User Defined Methods, and User Defined Types. It also offers Profiling and Lineage, showcasing column-level lineage between views and sourced columns, plus lineage for stored procedures. Users can track ownership and freshness of Databases and Tables, which helps understand data quality. Users can also see metadata about how the data was queried via SQL procedures, user defined functions and methods, and triggers, boosting trust in data products.

Start Exploring Today

These collectors enhance how data users explore, discover, understand, and trust their data. Whether you're a tech pro or not, these tools make navigating metadata easier and help teams become more data-driven. Dive into your data world with these new collectors, and embark on a journey of empowered decision-making.

Happy exploring, The data.world Team