We are thrilled to announce the General Availability of the Metadata Collectors page and CLI Command Builder tool! In addition, we've introduced the ability for users to create, manage, and delete Service Account tokens. These 3 features empower catalog administrators to more quickly set up on-premises collectors so your catalog users can get started discovering and understanding your data faster. In addition, seeing all the collectors (on-premises or cloud) that are bringing metadata into their catalogs allows you to maintain and govern your catalog more effectively.

For more information on these features, continue reading below.

Metadata Collectors Page: found in the Settings tab of an Organization, this page shows all of the collectors that are currently appearing in your catalog and other important information, such as the last time the collector ran. This page also includes cloud collectors set up via Connection Manager. For more information, refer to the documentation.

The CLI Command Builder allows users to step through a wizard to set up on-premises collectors. The wizard generates either a CLI command or a YAML file, so users can more quickly set up collectors during implementation. Since the BETA release, we've streamlined the form fields to more clearly differentiate required fields from optional fields For more information, refer to the documentation (available sources are denoted as "collector wizard available").

Service Accounts: administrators can now create, refresh (edit the expiration date), and delete service accounts from the UI. From the wizard, there is a "Create a service account" link that will take you to the "Service accounts" tab in the Settings page, and clicking on the "Add service account" button will generate an API token. We recommend using service accounts when setting up a collector, so the configurations aren't tied to user accounts. For more information, refer to the documentation.

We’re excited to announce new enhancements to data.world’s Databricks Collector and a brand new Collector for Azure Data Lake Storage Gen 2! With the help of these additional metadata harvesting and lineage capabilities, you can now get more detailed insights into your data than ever before.

Our Databricks Collector allows you to quickly and easily collect metadata from your Databricks environment into data.world. Now, with the addition of Jobs harvesting and lineage capabilities, you can get a deeper understanding of where your data is coming from, how it’s being used, and what insights you can discover.

Our new Jobs harvesting feature allows you to collect additional information about your workflows, such as creator, description, success, schedule, and more. This lets you better understand how and why your data was transformed.

The new lineage capabilities let you track your data’s journey, from its source all the way through its transformations. This means you can easily trace your data’s history, identify potential bottlenecks or sources of errors, and quickly gain an understanding of how your data has changed over time.

Our Azure Data Lake Storage Gen 2 Collector allows you to bring insights about your data storage layer into data.world. With this Collector, you can efficiently harvest metadata about Blobs and Containers, including the owner, last modified, path, and more. This information is vital for understanding your underlying data, leading to more trust and confidence in your data-driven decision-making.

You can learn more about these Features in our Databricks documentation and our Azure Data Lake Storage documentation. Both these Collectors are Tier 2 for Enterprise Customers.

An example of ADLS Blob metadata in the data.world platform

For customers with the data.world Standard Tier or above, two new tables are available in the standard events and logging package `baseplatformdata` dataset.  data.world has always provided metrics and the raw events that drive them as query dataset within the platform allowing our customers to build their own custom success measures, KPIs and dashboards for catalog adoption and metadata programs.  This capability now extends to metadata governance programs as well with special audit event log tables specifically designed to track changes to metadata.

Audit events allow administrators to monitor the actions performed by all the users in the data.world application through the UI or while using API. The audit log reporting functionality enhances the accountability of actions in the application. Administrators can track the actions taken by users in the application and find the root cause of issues by identifying the resources on which the action was performed and who performed the action.

Please see the Audit Events documentation for more details including full table descriptions and sample queries.

Last month, we introduced the new CLI Command Builder tool, which allows users to set through a wizard to generate a CLI command or YAML file to run a collector. Today, we're thrilled to announce that the CLI Command Builder supports all on-premises collectors.

We are happy to announce the release of our latest feature: Advanced Search within the global search bar. 

But wait, there's more!

For customers who have hierarchical collections, you'll want to check out our beta release of the collection picker in the Advanced Search modal. This gives users the ability to quickly scope their search to a branch of collections in the domain hierarchy. Be sure to turn on the beta feature flag in the advanced settings to see this feature.
 

Organization admins will love our new Groups access summary feature that gives them a full view of the various access a Group might have - organization-wide or direct access to collections and data workspaces (datasets and projects).

To find this new view, go to your Organization and click on Members > Groups > Access summary

You will see two sections: 1) organization-wide group access provides the group access to the entire catalog or all of the workspaces beyond the member default and 2) the direct access control section provides a view of the access the Group has to individual collections or workspaces along with the level of access to each. Users can manage direct access from this view without having to visit each collection or workspace.

This new view makes Group access management much easier by providing a one-stop summary of Group access. You can read more about managing Groups in our documentation portal.

At data.world, our goal is to help organizations unlock the full potential of their data. If you're interested in learning more about agile data governance, please visit our website and book a demo or reach out to your customer service representative.  We look forward to helping you manage your data and transform the way your users discover it!

The data.world enterprise catalog provides a 360-degree view of your data resources and semantics. Available in preview today, we've added quick section navigation links on the overview tab of your collection, resource, and glossary pages and separated related resources into their own, sortable, searchable views.

With this upcoming change, users will be able to scan the available metadata and get the insights needed to make decisions efficiently and effectively.

These views offer a highly organized and condensed presentation of the metadata, making it easier to quickly access and understand the information.

We'd love your feedback and thoughts before we roll them into the main UI. To see the views, please click "Turn on preview" in the banner at the top of any collection, resource, or glossary term page. To leave your feedback, please visit the help section (question mark in the lower left of the global navigation) and leave a suggestion via the support link.

You can read more about these changes in our documentation portal. If you're interested in learning more about our data discovery solutions, please visit our website and book a demo or reach out to your customer service representative today. We look forward to helping you and your teams discover your data!

We are pleased to release enhancements to our BigQuery Collector! Now, you can harvest column-level lineage between views and tables, as well as more metadata about datasets, projects, tables, and views.

These enhancements will enrich your data discovery experience, helping you understand your BigQuery data better. For instance, now you can use the Explorer Lineage interface to view lineage relationships between tables and views to track data flows. New metadata highlights include Dataset Labels, Last Modified, Date Modified, Created Date, Table Partitions, and View SQL. 

You can see the full list of harvested metadata in the documentation. As always, please reach out it you have questions!

Over the past several weeks, we've introduced a set of search IMPROVEMENTS we want to share:

1. Partial title search. Allows users to search for resources by entering just a portion of the title (3+ characters), making it easier to find the right data.
2. More related metadata search. From the context of a resource page, this improvement allows users the ability to search and filter related resources based on all the searchable metadata fields of a resource - including custom fields - which means it is now easier to filter large lists.
3. More camel case support. We have extended camel case support to our relationship filters. This makes it easier to find resources that have complex names that combine uppercase and lowercase letters.
4. Updated column search cards. This update improves the column search experience by providing users with additional information about columns such as database and datatype, making it easier to understand what each resource is without clicking through and back between the detail pages.
   
   
   

At data.world, our goal is to help organizations unlock the full potential of their data. We're constantly improving search in order to better serve our customers looking to take data management and discovery to the next level.

If you're interested in learning more about our data discovery solutions, please visit our website and book a demo. You can also read more about our search features on the docs portal. We look forward to helping you manage your data and transform the way your users discover it!

We are excited to begin rolling out for preview some exciting ENHANCEMENTS to the user experience on our collections, metadata resources, and glossary pages.

Today, you'll notice a new PREVIEW button on these pages. Click on it to get a preview of some of our latest features.

feature 

• Metadata sections navigation - a table-of-contents-like side menu for easy access to your metadata sections, related resources, etc.
• Collection hierarchy widget - a navigable tree of your data taxonomy.

COMING SOON 

• Relationships UX improvements  - a more information-rich view of the related resources, improved edit/suggest flows.
• Custom icons - dress your custom types in attire that makes sense to you and your catalog users.

To find out more about these new navigation features, please visit our documentation portal.