January Releases: Expanding Connectivity, Simplifying Setup, and Enhancing Lineage Exploration

This month, we’re delivering powerful updates to enhance metadata collection, streamline setup, and improve lineage exploration. We’re expanding our connectivity with new on-premise collectors for Apache Airflow and Qlik Talend, enabling deeper metadata harvesting and lineage tracking for critical ETL and workflow automation tools. People field configuration is now more intuitive, allowing user accounts to be dynamically selected for ownership and stewardship, reducing manual setup and improving governance. Finally, our new public API endpoint for lineage querying makes it easier for customers to customize lineage exploration with flexible queries and standardized outputs. These updates help teams work smarter, get to insights faster, and build on top of our platform with greater ease. 🚀

Support for Airflow and Talend on-premise Collection

By the end of January, the data.world collector integrations will include new collectors for Apache Airflow and Qlik Talend Data Integration (the on-premise version of Qlik’s Talend product). Airflow is an open source workflow automation tool that many enterprises use to schedule and manage data engineering and analytical tasks. The new collector will harvest metadata about these workflows–called Directed Acyclic Graphs–and the tasks contained within them. Qlik Talend is a data integration product that facilitates extract, transform, and load (ETL) processes; the new collector will identify sources and targets of these processes and harvest lineage relationships representing the flow of data between them.

These collectors will initially be available as on-premise collectors only, but will also be available as cloud collectors in early February.

Streamlined Setup of People Fields

Configuring ownership and stewardship just got easier! In addition to supporting people as collected resources, customers can now utilize their user accounts to populate people fields. This update streamlines setup, providing an intuitive approach that helps teams quickly setup, ensuring seamless attribution and governance from the start and helping end-users connect with the right people. You can read the documentation for this feature here.

Resource Lineage Support in Public API

We’re making it easier than ever to programmatically explore lineage with our new Catalog Lineage Public API endpoint! This update provides flexible query options, allowing customers to tailor lineage exploration to their needs to build lineage based tooling, automations, and integrations. This is a win for all lineage customers looking for deeper insights and more intuitive ways to navigate their data relationships. 

UX Changes Coming Soon

Activity feed for Resources

Soon, we’ll be introducing a new Activity tab on Resources, Glossary and Collection objects that show edit history and other activity in the UI. This will make it easier for users to quickly understand how the resources have been updated and changed. Announcement of the release will soon follow.

Resource page redesign

Along with a new activity feed, we'll be introducing a newly designed details page that offers more intuitive navigation, better use of whitespace, configurable relationship tabs, inline editing, and other features that will make the resources both easier to understand and scan but also easier to enrich and edit. Announcement of this release will follow in this quarter.