We've been rolling out a series of features to make working with collections easier and more intuitive on data.world. In case you missed it, here a few recent collections related updates:

1. New Collection Management API
2. Quick Collection Creation
3. Editing Resource Collections
4. Filtering Collections In Search

Most recently, we've released bulk editing collections, which allows you to quickly manage a large number of resources by adding and removing them from collections with just a few clicks. 

Screenshot Walk-through

Using business terms as our example, we'll start off by clicking the "triple dot" menu and selecting "Edit multiple Business terms":

Select all the resources you want to edit:

Chose "Add to existing" to add the resources to an additional collection:

Review your changes:

After confirming, you'll see a message like this:

Quick Video Demo

This is a default behavior change that shows more metadata fields while in edit mode. This makes it easier to see the fields that are available for annotating, documenting, and governing catalog resources.

If you have any concerns or questions on how to manage field visibility, please reach out to your customer success representative or support@data.world and we'd be happy to assist!

For more information on this feature, see this Coming Soon post related to the change: https://whatsnew.data.world/coming-soon:-changes-to-custom-metadata-field-configurations-2UHSOA

data.world now sends 2 additional webhooks: Dataset Sync Failure and Dataset Sync Recovery Success. 

When a dataset within your org fails to sync, data.world will immediately send a webhook message to the configured URL with details about the failure, including error code and error messages from the downstream source.

With those sync failure details, you could configure a ticket to be created in your ticketing system, trigger an alert for a dev-ops team, or simply just make the team aware of the issue.

Natively, data.world will format the webhook payload to post messages in Slack when it recognizes the configured webhook is Slack, which is shown in the demo video below.

Demo

Screenshots

Technical Details

syncTask.status.success is triggered only after the recovery from a failure. 

Example payload, in JSON:

{
  "datasetid": "sync-demo",
  "event": {
    "type": "SUCCESS",
    "task": "state_table.csv",
    "created": "2021-02-04T06:37:06.936Z"
  },
  "eventType": "syncTask.status.success",
  "resourceUrl": "https://data.world/texas-whiskey/sync-demo/file/state_table.csv",
  "resourceType": "URL",
  "resourceOwner": "texas-whiskey"
}

 syncTask.status.failure is triggered on the first failure for that file. When it recovers, data.world will send one syncTask.status.success

Example payload, in JSON:

{
  "datasetid": "sync-demo",
  "event": {
    "type": "FAILURE",
    "task": "state_table.csv",
    "created": "2021-02-04T06:35:17.945Z",
    "error": "HTTP/1.1 404 Not Found"
  },
  "eventType": "syncTask.status.failure",
  "resourceUrl": "https://data.world/texas-whiskey/sync-demo/file/state_table.csv",
  "resourceType": "URL",
  "resourceOwner": "texas-whiskey"
}

We're proud to announce that a new relational database metadata collector is now available for enterprise customer use -- SAP SQL Anywhere (formerly known as Sybase SQL Anywhere), versions 16 & 17.

The following update will only be available for Sites customers:

Managing data access requests takes a team - and data.world wants to make it easier for your data access request team, whether it be your data stewards, owners, or stakeholders, to make informed and timely decisions about each request.

Previously, each of the approving parties for a data access request would received a discrete, standalone email informing them of the data access request, including the customizable request access form details the user filled out.

Now, for Sites customers, we'll be sending out a single email to the approving parties, with the requesting user copied on the email.

This provides the requesting user a channel to engage further with the approving team, as well as a method for the approving team to communicate amongst themselves before rendering a decision about the request.

Real world example:

Let's say you have 3 administrators on the dataset, and a user requests access to the data. Previously, we'd send out 3 individual emails to each of the 3 administrators. Aside from the recipient, the emails are identical.

With our changes, we'll send out just a single email, but to 4 recipients: the 3 administrators, and the requesting user. Any of those users can reply-all, remove recipients, or add recipients in your preferred email client.

Have you ever need to create a 7-day trailing average in order to smooth out a graph? How about compare a value to the same value in the previous row?

Windowed aggregations are a powerful way to perform complex analysis, such as this, without leaving the comfort of the SQL. These functions operate on a defined group, or "window", of rows to return an aggregated value for each row. Window aggregations are supported for both uploaded tabular data, as well as live tables (where support exists in the connected database).

Note: this feature is currently in Beta. Please contact support with any issues.

Keep your eyes open for an exciting new feature in upcoming enterprise releases.
Soon, our standard search behavior will be expanded to include custom metadata associated with your catalog resources. This feature will enable users to execute generic searches such as "Smith" and find resources where "Jane Smith" is listed as "Steward" (a custom metadata property) for a given resource. This feature will be accessible to enterprise customers and will include public API support.

Today we released a new filter group for the search page. Customers are using collections to group and organize the items within their catalog. Users can now filter their search results based on the catalog collection an item belongs to.
This feature is visible to enterprise customers.

Head over to our api documentation to learn more about the new API endpoints we've added in support of collection management.

You can now create and manage collections from the public API to help build automation pipelines and integrations around your enterprise data catalog.

Our interactive docs allow you to test requests right from the browser.

 🚨 Default behavior change coming soon! 🚨

data.world offers the ability to define custom metadata fields on resources like datasets, projects, glossary terms, dashboards, and metadata tables. Custom metadata field types include free text fields, drop-down fields, and multi-select hierarchical fields.

To provide a focused user experience while editing custom metadata, each field had a configuration option to either show the field during the editing process, or else put the field behind a drop-down, requiring a user to find the field in the drop-down before setting its value. The default behavior was to hide the field in the "Add new field" drop-down (pictured below), unless it was flagged as a primary field.

While we typically avoid changing the default behavior of features with widespread adoption, conversations with customers has led us to flipping the default — all fields will show up during the editing experience, unless they are now flagged a secondary field. Only if they are considered a secondary field will they ever be behind the "Add new field" drop down.

The benefit of this change is that users will more easily see all the fields available to edit or provide crowdsourced suggestions for, including fields that are empty and don't have a value set yet.

The edit experience, along with the "Add new field" drop down, in action:

Bottom line - this will result in more fields showing up by default when you click Edit (they won't be hidden under the 'Add new field' drop-down) - unless you mark them as secondary in your metadata configuration (which will hide them under the drop-down).

If you have any concerns or questions on how to implement the secondary setting, including if you'd like to implement it proactively prior to the feature being rolled out, please reach out to your customer success representative or support@data.world and we'd be happy to assist!