Announcing support for Confluent Kafka metadata

Announcing our newest metadata collector - Confluent Kafka! We know how important it is to have the most up-to-date streaming data, so we’ve created this collector to allow you to easily monitor and collect Kafka metadata from your Confluent streaming platform. 

With Kafka metadata in data.world, you and your teams can: 

  • Easily discover and monitor streaming metadata for real-time applications
  • Understand what is being streamed from on-prem and cloud Confluent
  • Have a single source of truth for your Confluent schemas for better discovery and governance

The data.world Confluent Collector is actually two collectors, one for Confluent Platform (on-prem) and one for Confluent Cloud. With these collectors, you can capture, store, and analyze metadata including Cluster, Consumer, Producer, Broker, Partition, Schema, Consumer Group, Topic, and Environment (for Cloud). The collectors can optionally harvest metadata from Avro, JSON-schema, and Protobuf schemas stored in Confluent Schema Registry.

These Collectors are Tier 2 for Enterprise Customers. You can read the full documentation for Confluent Platform here and for Confluent Cloud here.

Avro Schema example metadata

An example of metadata for an Avro Schema in the data.world platform

Announcing Azure Data Lake Storage Gen 2 Collector and Databricks Collector Lineage and Jobs

We’re excited to announce new enhancements to data.world’s Databricks Collector and a brand new Collector for Azure Data Lake Storage Gen 2! With the help of these additional metadata harvesting and lineage capabilities, you can now get more detailed insights into your data than ever before.

Our Databricks Collector allows you to quickly and easily collect metadata from your Databricks environment into data.world. Now, with the addition of Jobs harvesting and lineage capabilities, you can get a deeper understanding of where your data is coming from, how it’s being used, and what insights you can discover.

Our new Jobs harvesting feature allows you to collect additional information about your workflows, such as creator, description, success, schedule, and more. This lets you better understand how and why your data was transformed.

The new lineage capabilities let you track your data’s journey, from its source all the way through its transformations. This means you can easily trace your data’s history, identify potential bottlenecks or sources of errors, and quickly gain an understanding of how your data has changed over time.

Our Azure Data Lake Storage Gen 2 Collector allows you to bring insights about your data storage layer into data.world. With this Collector, you can efficiently harvest metadata about Blobs and Containers, including the owner, last modified, path, and more. This information is vital for understanding your underlying data, leading to more trust and confidence in your data-driven decision-making.

You can learn more about these Features in our Databricks documentation and our Azure Data Lake Storage documentation. Both these Collectors are Tier 2 for Enterprise Customers.

An image showing am Blob from ADLS in the data.world platform

An example of ADLS Blob metadata in the data.world platform


data.world Usage and Audit Events now available as a Snowflake Marketplace Private Listing

As a Snowflake Powered By partner, data.world is proud to announce that usage and audit event data, previously only available in a data.world dataset, is now available as a Snowflake Marketplace Private Listing.  This allows data.world customers access to their full history of data.world events data via the Snowflake Data Cloud enabling high performance and advanced analytic functions on this data.  It also makes data.world events and logging data available via Snowflake with no ETL required for integration in a wide variety of use cases.  To read more about this capability and how to request access to a Private Listing, please see our documentation here.

New Events Available for Metadata Audit

For customers with the data.world Standard Tier or above, two new tables are available in the standard events and logging package `baseplatformdata` dataset.  data.world has always provided metrics and the raw events that drive them as query dataset within the platform allowing our customers to build their own custom success measures, KPIs and dashboards for catalog adoption and metadata programs.  This capability now extends to metadata governance programs as well with special audit event log tables specifically designed to track changes to metadata.

Audit events allow administrators to monitor the actions performed by all the users in the data.world application through the UI or while using API. The audit log reporting functionality enhances the accountability of actions in the application. Administrators can track the actions taken by users in the application and find the root cause of issues by identifying the resources on which the action was performed and who performed the action.

Please see the Audit Events documentation for more details including full table descriptions and sample queries.

CLI Command Builder now supports more on-premises collectors

Last month, we introduced the new CLI Command Builder tool, which allows users to set through a wizard to generate a CLI command or YAML file to run a collector. Today, we're thrilled to announce that the CLI Command Builder supports all on-premises collectors.

With this feature, catalog administrators can more easily set up on-premises collectors during implementation. Once you've run a collector, you can see all the collectors (on-premises or cloud) that are bringing metadata into your catalog on the Metadata Collectors page.

For more information on the CLI Command Builder, refer to the documentation for each collector ("Metadata collector" column).

Resource page navigation just got easier!

Our resource pages just got easier to quickly scan and navigate. With the latest update now in production, the changes mentioned in our earlier post, provide users with the ability to scan the available metadata and get the insights needed to make decisions efficiently and effectively.

The quick section navigation links on the overview tab of your collection, resource, and glossary pages will make navigating the areas of the resource page faster. 


We've also separated the overview and the details of the different types of related resources by giving the related resources their own tabs. The related resources are now highly scannable, sortable, expandable, and searchable. These views offer an organized and condensed presentation of the metadata, making it easier to quickly access and understand the information.


As always, you can read more about these changes in our documentation portal. If you have feedback, please leave a suggestion in our support portal. For those interested in learning more about our data discovery solutions, please visit our website and book a demo or reach out to your customer service representative today. We look forward to helping you and your teams discover your data!

Advanced Search now in the global search bar

We are happy to announce the release of our latest feature: Advanced Search within the global search bar. 


In addition to adding Advanced Search to our global search bar, we've improved it by including the ability to pre-scope your search to a certain Organization. If you are an Enterprise customer with multiple organizations, you can now pre-select the organization to which you want to limit your search.

But wait, there's more!

For customers who have hierarchical collections, you'll want to check out our beta release of the collection picker in the Advanced Search modal. This gives users the ability to quickly scope their search to a branch of collections in the domain hierarchy. Be sure to turn on the beta feature flag in the advanced settings to see this feature.
 


The new advanced search features in the global search bar are designed to provide a more intuitive and powerful data discovery experience from login. With these new capabilities, you'll be able to find the data and insights you need, faster than ever before.

Refer to the advanced search documentation to learn more.

We're committed to continually improving our platform and are eager to hear your feedback. Please feel free to reach out via our support portal to share your thoughts, experiences, and suggestions regarding our new Advanced Search features. Visit our website to learn more about Data Discovery or book a demo. Together, let's unlock the true potential of data-driven decision-making.

Bulk upload and edit all metadata resources

We're thrilled to bring a much-anticipated feature to the platform: the ability to bulk edit all metadata resources by Collection!

Users could previously bulk load and edit business glossary terms, but now this is possible for any type of resource. Simply download a spreadsheet of all the resources in a Collection, enrich them by editing fields or add new resources, and upload the changes to the platform.

On the Settings page of a Collection, users with edit access to the Collection will see the following modal:

Clicking "Download resources" will generate a spreadsheet that contains all resources, sorted by resource type, in that collection. This spreadsheet contains helpful instructions on how to use the spreadsheet on the "Overview" sheet.


After making changes, users can upload the spreadsheet and see a summary of the changes made before confirming the upload:


For more information, refer to the documentation.

Groups access management just got easier!

Organization admins will love our new Groups access summary feature that gives them a full view of the various access a Group might have - organization-wide or direct access to collections and data workspaces (datasets and projects).

To find this new view, go to your Organization and click on Members > Groups > Access summary


You will see two sections: 1) organization-wide group access provides the group access to the entire catalog or all of the workspaces beyond the member default and 2) the direct access control section provides a view of the access the Group has to individual collections or workspaces along with the level of access to each. Users can manage direct access from this view without having to visit each collection or workspace.


This new view makes Group access management much easier by providing a one-stop summary of Group access. You can read more about managing Groups in our documentation portal.

At data.world, our goal is to help organizations unlock the full potential of their data. If you're interested in learning more about agile data governance, please visit our website and book a demo or reach out to your customer service representative.  We look forward to helping you manage your data and transform the way your users discover it!

Introducing the Metadata Collectors Page and CLI Command Builder

Introducing the new Metadata Collectors page and CLI Command Builder tool!

With these features, catalog administrators can more easily set up on-premises collectors and see all the collectors (on-premises or cloud) that are bringing metadata into their catalogs.

Currently in BETA, this feature has two components:

  • CLI Command Builder: allows users to step through a wizard to set up 16 on-premise collectors: Athena, AWS Glue, Databricks, Db2, dbt, MySQL, Oracle, PostgreSQL, PowerBI, PowerBI Gov, Presto, RedShift, Snowflake, SqlAnywhere, SQLServer, Vertica. The wizard generates either a CLI command or a YAML file, so users can more quickly set up collectors during implementation. For more information, refer to the documentation (available sources are denoted as "collector wizard available"). In an upcoming release, all on-premise collectors will be available.

  • Metadata Collectors Page: found in the Settings tab of an Organization, this page shows all of the collectors that are currently appearing in your catalog and other important information, such as the last time the collector ran. This page also includes cloud collectors set up via Connection Manager. For more information, refer to the documentation.

Show Previous EntriesShow Previous Entries