January Releases: Expanding Connectivity, Simplifying Setup, and Enhancing Lineage Exploration

This month, we’re delivering powerful updates to enhance metadata collection, streamline setup, and improve lineage exploration. We’re expanding our connectivity with new on-premise collectors for Apache Airflow and Qlik Talend, enabling deeper metadata harvesting and lineage tracking for critical ETL and workflow automation tools. People field configuration is now more intuitive, allowing user accounts to be dynamically selected for ownership and stewardship, reducing manual setup and improving governance. Finally, our new public API endpoint for lineage querying makes it easier for customers to customize lineage exploration with flexible queries and standardized outputs. These updates help teams work smarter, get to insights faster, and build on top of our platform with greater ease. 🚀

Support for Airflow and Talend on-premise Collection

By the end of January, the data.world collector integrations will include new collectors for Apache Airflow and Qlik Talend Data Integration (the on-premise version of Qlik’s Talend product). Airflow is an open source workflow automation tool that many enterprises use to schedule and manage data engineering and analytical tasks. The new collector will harvest metadata about these workflows–called Directed Acyclic Graphs–and the tasks contained within them. Qlik Talend is a data integration product that facilitates extract, transform, and load (ETL) processes; the new collector will identify sources and targets of these processes and harvest lineage relationships representing the flow of data between them.

These collectors will initially be available as on-premise collectors only, but will also be available as cloud collectors in early February.

Streamlined Setup of People Fields

Configuring ownership and stewardship just got easier! In addition to supporting people as collected resources, customers can now utilize their user accounts to populate people fields. This update streamlines setup, providing an intuitive approach that helps teams quickly setup, ensuring seamless attribution and governance from the start and helping end-users connect with the right people. You can read the documentation for this feature here.

Resource Lineage Support in Public API

We’re making it easier than ever to programmatically explore lineage with our new Catalog Lineage Public API endpoint! This update provides flexible query options, allowing customers to tailor lineage exploration to their needs to build lineage based tooling, automations, and integrations. This is a win for all lineage customers looking for deeper insights and more intuitive ways to navigate their data relationships. 

UX Changes Coming Soon

Activity feed for Resources

Soon, we’ll be introducing a new Activity tab on Resources, Glossary and Collection objects that show edit history and other activity in the UI. This will make it easier for users to quickly understand how the resources have been updated and changed. Announcement of the release will soon follow.

Resource page redesign

Along with a new activity feed, we'll be introducing a newly designed details page that offers more intuitive navigation, better use of whitespace, configurable relationship tabs, inline editing, and other features that will make the resources both easier to understand and scan but also easier to enrich and edit. Announcement of this release will follow in this quarter.

The October release of data.world brings a wide variety of new capabilities and improvements across the platform – read on to learn more about the GA of Databricks Publisher, new collectors for MongoDB and Alteryx, Okta support in SCIM, the GA of the improved search experience, and more!

Additionally, we highlight some changes made to the data.world Open Data Community to improve privacy and preserve the quality of open data and the user experience.

Databricks Publisher Premium Automation

We’re excited to announce the GA launch of the Databricks Publisher Premium Automation! This new feature allows users to seamlessly publish metadata from data.world to Databricks, simplifying the process of managing and synchronizing key data attributes. Specifically, users can now automatically publish table and column descriptions from data.world to Databricks and push selected metadata attributes as Databricks tags. Whether you prefer manual updates or fully automated syncing, this automation ensures that metadata remains consistent between platforms, reducing manual effort and improving data integrity. With data.world now acting as the source of truth, your metadata stays up-to-date across systems effortlessly.

For more information, see the product documentation.

New MongoDB and Alteryx Collectors

This month, we’re excited to announce new MongoDB and Alteryx Collectors, both available in Private Preview. If you’re interested in early access to either of these new collectors, please reach out to your Customer Service Director.

MongoDB Collector

The MongoDB Collector catalogs metadata from MongoDB, helping maintain a comprehensive inventory of MongoDB assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for MongoDB databases, collections, views, indexes and more.

An example collection from MongoDB

Alteryx Collector

The Alteryx Collector catalogs metadata from Alteryx, helping maintain a comprehensive inventory of Alteryx assets, facilitating better governance, discovery, and utilization of data across your organization.

This collector harvests metadata for workflows, workflow nodes, workflow jobs, connections, schedules and more.

An example collection from Alteryx

Improved lineage for SQL Server

The SQL Server Collector now collects additional lineage relationships not previously captured through SQL parsing using built-in SQL Server functions that describe relationships between objects (such as, in some cases, the columns and tables referenced by views or stored procedures).

For more information and detail, see the description of lineage collected by the SQL Server Collector in the product documentation.

Support for Okta in SCIM

The active Private Preview of SCIM (System for Cross-domain Identity Management) now additionally supports Okta (in addition to Microsoft Entra ID), allowing customers who use Okta as their enterprise identity provider to have automated management of users and groups in data.world.

If you are interested in being part of the SCIM Private Preview, or just want to learn more, please reach out to your Customer Service Director.

Webhook Authorization enhancement

Webhooks now support an optional authorization key parameter to help consuming applications verify the origin and permissions for an incoming webhook. Learn more

Collection Details in Technical Reference

By popular request, the Technical Reference page for catalog resources now includes details about the collections the resource belongs to. Learn more

Relative Time Advanced Search Syntax

Create powerful saved searches for resources by updated and created dates using three new relative time options:

• `created:today` 
• `updated:yesterday`
• `created:{last 30 days}`

See the product documentation on creating advanced searches to learn more.

UX Improvements

Adding the new search experience to Organizations: Our new search experience has been a big hit with users. It’s faster, cleaner, and provides more advanced features in the UI. We've fully retired the classic experience and brought the new search features to the Resources, Glossary and Collection landing pages.

Coming Soon! Advanced relationship editing: We're adding new improvements that make it easier to find the right resources and add or remove more than one relationship at a time.

Coming Soon! More look-and-feel updates: Next up in our work to update and modernize our UI, we'll be swapping the old default avatars to a newer color palette and default avatar design that utilizes letters. This change will also provide a more accessible experience as it gives users the ability to distinguish users and organizations using letters.

Changes to data.world Open Data Community

data.world Open Data Community profiles, datasets, and projects now behind a login wall: To better control the privacy of our users and to protect the effectiveness of the content on our active Open Data Community, we have made the decision to restrict access to profiles, datasets, and projects to account holders. It is always free to join our open data community. 

data.world Open Data Community commenting restrictions: Commenting is now restricted to contributors on datasets and projects in the data.world Open Data Community. Organizations can enable comments on their public datasets through organization settings. This feature is not available or enforced for Enterprise customers on private instance or VPC deployments.

Visit your notifications settings page to customize the transactional emails you receive from data.world.

You can choose to:

• Turn off all non-essential email communications
• Unsubscribe from a category of email notifications
• Customize which digests you receive
• Customize dataset and project activity notifications

Learn more

We are happy to announce the release of our latest feature: Advanced Search within the global search bar. 

But wait, there's more!

For customers who have hierarchical collections, you'll want to check out our beta release of the collection picker in the Advanced Search modal. This gives users the ability to quickly scope their search to a branch of collections in the domain hierarchy. Be sure to turn on the beta feature flag in the advanced settings to see this feature.
 

The data.world enterprise catalog provides a 360-degree view of your data resources and semantics. Available in preview today, we've added quick section navigation links on the overview tab of your collection, resource, and glossary pages and separated related resources into their own, sortable, searchable views.

With this upcoming change, users will be able to scan the available metadata and get the insights needed to make decisions efficiently and effectively.

These views offer a highly organized and condensed presentation of the metadata, making it easier to quickly access and understand the information.

We'd love your feedback and thoughts before we roll them into the main UI. To see the views, please click "Turn on preview" in the banner at the top of any collection, resource, or glossary term page. To leave your feedback, please visit the help section (question mark in the lower left of the global navigation) and leave a suggestion via the support link.

You can read more about these changes in our documentation portal. If you're interested in learning more about our data discovery solutions, please visit our website and book a demo or reach out to your customer service representative today. We look forward to helping you and your teams discover your data!

Over the past several weeks, we've introduced a set of search IMPROVEMENTS we want to share:

1. Partial title search. Allows users to search for resources by entering just a portion of the title (3+ characters), making it easier to find the right data.
2. More related metadata search. From the context of a resource page, this improvement allows users the ability to search and filter related resources based on all the searchable metadata fields of a resource - including custom fields - which means it is now easier to filter large lists.
3. More camel case support. We have extended camel case support to our relationship filters. This makes it easier to find resources that have complex names that combine uppercase and lowercase letters.
4. Updated column search cards. This update improves the column search experience by providing users with additional information about columns such as database and datatype, making it easier to understand what each resource is without clicking through and back between the detail pages.
   
   
   

At data.world, our goal is to help organizations unlock the full potential of their data. We're constantly improving search in order to better serve our customers looking to take data management and discovery to the next level.

If you're interested in learning more about our data discovery solutions, please visit our website and book a demo. You can also read more about our search features on the docs portal. We look forward to helping you manage your data and transform the way your users discover it!

We are excited to begin rolling out for preview some exciting ENHANCEMENTS to the user experience on our collections, metadata resources, and glossary pages.

Today, you'll notice a new PREVIEW button on these pages. Click on it to get a preview of some of our latest features.

feature 

• Metadata sections navigation - a table-of-contents-like side menu for easy access to your metadata sections, related resources, etc.
• Collection hierarchy widget - a navigable tree of your data taxonomy.

COMING SOON 

• Relationships UX improvements  - a more information-rich view of the related resources, improved edit/suggest flows.
• Custom icons - dress your custom types in attire that makes sense to you and your catalog users.

To find out more about these new navigation features, please visit our documentation portal.

We have some awesome news! Our member access summary page is now live!

The member access summary makes it much easier for members to understand the various levels of access they have in an organization. As we work to make our access levels more flexible and granular, this page also gives our org admins the ability to quickly audit the level of access their members have and take action.

With our latest release, users are now able to bookmark even more things. Just click on the bookmark icon from search results on the Resources tab or directly on the detail page to create bookmarks for your metadata resources, insights, collections, datasets, projects and more.

As you click on the bookmark you'll see how many other people have bookmarked the page. Find the full list of your bookmarks on your bookmark page and your latest additions on your personal Action Center home page.

Visit our Docs Portal for more details about bookmarks.

We're excited to announce that we've released a significant new feature to give organizations more flexibility in managing members and permissions through User Groups.

With groups, you can:

• Create new custom groups to independently manage people, data, and metadata on the platform much more easily.
• Grant groups access to organization-owned datasets and projects.
• Manage different levels of access to all metadata catalog resources via groups.
• Have different access levels for people responsible for catalog configuration and for catalog curation.

This feature has changed how new users are added to organizations on the platform. This short video highlights the changes.