Introducing the new Metadata Collectors page and CLI Command Builder tool!

With these features, catalog administrators can more easily set up on-premises collectors and see all the collectors (on-premises or cloud) that are bringing metadata into their catalogs.

Currently in BETA, this feature has two components:

• CLI Command Builder: allows users to step through a wizard to set up 16 on-premise collectors: Athena, AWS Glue, Databricks, Db2, dbt, MySQL, Oracle, PostgreSQL, PowerBI, PowerBI Gov, Presto, RedShift, Snowflake, SqlAnywhere, SQLServer, Vertica. The wizard generates either a CLI command or a YAML file, so users can more quickly set up collectors during implementation. For more information, refer to the documentation (available sources are denoted as "collector wizard available"). In an upcoming release, all on-premise collectors will be available.

• Metadata Collectors Page: found in the Settings tab of an Organization, this page shows all of the collectors that are currently appearing in your catalog and other important information, such as the last time the collector ran. This page also includes cloud collectors set up via Connection Manager. For more information, refer to the documentation.

The data.world enterprise catalog provides a 360-degree view of your data resources and semantics. Available in preview today, we've added quick section navigation links on the overview tab of your collection, resource, and glossary pages and separated related resources into their own, sortable, searchable views.

With this upcoming change, users will be able to scan the available metadata and get the insights needed to make decisions efficiently and effectively.

These views offer a highly organized and condensed presentation of the metadata, making it easier to quickly access and understand the information.

We'd love your feedback and thoughts before we roll them into the main UI. To see the views, please click "Turn on preview" in the banner at the top of any collection, resource, or glossary term page. To leave your feedback, please visit the help section (question mark in the lower left of the global navigation) and leave a suggestion via the support link.

You can read more about these changes in our documentation portal. If you're interested in learning more about our data discovery solutions, please visit our website and book a demo or reach out to your customer service representative today. We look forward to helping you and your teams discover your data!

We're happy to announce a NEW FEATURE called Collection hierarchy. This feature will help our enterprise customers organize their domain-driven data catalogs to simplify data management and improve data discovery. 

Collection hierarchy is a tree-like view of your hierarchical collection relationships viewable on the collection overview tab. It organizes data resources and semantic concepts into increasingly more granular or specific groups based on their common characteristics - like domains, categories, markets, etc. It allows you to express your data taxonomy in a way that makes sense to your users:

• Easily find relevant data: With resources and terms grouped into collections, users can quickly navigate relevant collections.
• Simplify data management: By grouping data assets into hierarchical collections, you can simplify the process of assigning manage and edit levels of the catalog metadata.

For example, a sales steward might organize resources and terms into different Sales subcollections, such as Sales by Region, Sales by Product, and Sales by Customer. With this structure, your sales analysts can easily find the data they need to make more informed decisions.

Collection hierarchy, combined with other recently released features like Groups, Collection Access Control, and the ability to create new Collection types, provides organizations with the building blocks to build powerful data products and organize their domain-driven data catalog, simplifying data management, and improving decision-making. We look forward to seeing the positive impact it will have on our customers.

At data.world, our goal is to help organizations unlock the full potential of their data. If you're interested in learning more about data mesh and our solutions, please visit our website and book a demo. You can also read more about our collection features on the documentation portal. We look forward to helping you manage your data and transform the way your users discover it!

Introducing our newest metadata collector: Amazon S3 🎉 

The Amazon S3 Collector catalogs buckets and objects, allowing you to quickly search and discover your data. This new collector harvests metadata about buckets and objects, including the Region, Version State, Size, Last Modified Data, ACL Owner, Grantee and Grant Permission, amongst others (see the full list in the documentation). 

Inside the data.world platform, users will be able to view the relationships between S3 buckets and Objects, enhancing data discoverability. Using our configurable UI, you can display which pieces of metadata are most important to you, such as ACL Permission or S3 Metadata Keys and Values.

Learn how to use the new Amazon S3 Collector in our documentation, or please reach out if you have questions!

Over the past several weeks, we've introduced a set of search IMPROVEMENTS we want to share:

1. Partial title search. Allows users to search for resources by entering just a portion of the title (3+ characters), making it easier to find the right data.
2. More related metadata search. From the context of a resource page, this improvement allows users the ability to search and filter related resources based on all the searchable metadata fields of a resource - including custom fields - which means it is now easier to filter large lists.
3. More camel case support. We have extended camel case support to our relationship filters. This makes it easier to find resources that have complex names that combine uppercase and lowercase letters.
4. Updated column search cards. This update improves the column search experience by providing users with additional information about columns such as database and datatype, making it easier to understand what each resource is without clicking through and back between the detail pages.
   
   
   

At data.world, our goal is to help organizations unlock the full potential of their data. We're constantly improving search in order to better serve our customers looking to take data management and discovery to the next level.

If you're interested in learning more about our data discovery solutions, please visit our website and book a demo. You can also read more about our search features on the docs portal. We look forward to helping you manage your data and transform the way your users discover it!

With the new year comes new features! We are pleased to launch our newest metadata capability: data profiling. This new feature creates metadata describing summary statistics for columns when a collector is run.

These summary statistics will help you understand and trust your data by providing a quick look at the data. For instance, viewing stats like the minimum and maximum values shows the shape of the data, allowing you to know quickly if your data is as expected. 

How can you create profiling metadata? This feature is currently available via the Snowflake, SQL Server, PostgreSQL, and Redshift collectors with more collectors on the near horizon. There are three optional commands that can be used during the collector run to generate the profiling metadata: 

--enable-column-statistics  description: enables harvesting of column statistics

--sample-string-values  description: enables harvesting of histograms for columns containing string data

 --target-sample-size  description: controls the number of rows sampled for computation of column statistics and string-value histograms

You can read more about these commands on the following collector documentation pages:

• Snowflake collector documentation
• SQL Server collector documentation
• PostgreSQL collector documentation
• Redshift collector documentation

We are excited to begin rolling out for preview some exciting ENHANCEMENTS to the user experience on our collections, metadata resources, and glossary pages.

Today, you'll notice a new PREVIEW button on these pages. Click on it to get a preview of some of our latest features.

feature 

• Metadata sections navigation - a table-of-contents-like side menu for easy access to your metadata sections, related resources, etc.
• Collection hierarchy widget - a navigable tree of your data taxonomy.

COMING SOON 

• Relationships UX improvements  - a more information-rich view of the related resources, improved edit/suggest flows.
• Custom icons - dress your custom types in attire that makes sense to you and your catalog users.

To find out more about these new navigation features, please visit our documentation portal.

We are very excited to announce our newest metadata collection feature: Snowflake Tags & Policy harvesting! 

The Snowflake collector can now harvest Snowflake object tags, Snowflake tag-based masking policies, and Snowflake row access policies. This new feature will enhance your data governance experience by allowing you to see if a tag or policy is applied to a table or column coming from Snowflake.

For instance, here is a screenshot from the data.world catalog showing how a Snowflake Tag-Based Masking Policy has been applied to sensitive data columns: routing numbers, bank name, and bank account number. In this view, you can also see the associated Tag (Classification:confidential), and technical details about the Policy, like the Policy Body which explains how the Policy works.

How can you use this new feature? There are 2 optional commands for harvesting this information within the Snowflake collector run. Read more about it in the Snowflake Collector documentation.

Stay tuned for more exciting governance features in the coming months!

Big news today! 🎉

You asked for more granular control of your data catalog and we listened. We're excited to introduce Collection Access Control. This NEW Feature is going to help you scale your Agile Data Governance program by providing more granular ways to control the access and management of your data catalog.

Collection Access Control provides role-based control to your metadata resources by collection, helping you target who can see and edit the resources in your catalog.

Read more about how to manage collection access on our documentation portal.

Custom resources play an important role in a data catalog's ability to accurately represent your company's metadata. Users need to be able to add resources that aren't directly from data sources to give a full picture of their data landscapes. Now, users can create these custom resources directly from the UI and manage them like any other resource.

After designating which resources should have this feature enabled in your metadata profile, users will be able to access these resources in the "Other resources" section in the "New resource" dropdown (in the example below, the custom resources are "Bank account" and Credit card").

For more information, refer to the documentation.