Following major releases at the end of 2023 for both Kedro and Kedro-Viz, we've kept the momentum going with eight more releases for kedro, fourteen for kedro-plugins, seven for kedro-viz and a brand-new project vscode-kedro ! Here, we highlight the ten most impactful new features introduced this year.
1. Enhanced catalog and dataset exploration functionality
We've made it easier than ever to explore your data with Kedro:
Pretty printing: both the DataCatalog and individual datasets now support pretty printing for better readability.
Key completion: easily access datasets within the DataCatalog using key completion.
These improvements significantly streamline how you interact with and manage your data catalog.
2. Debug nodes inside notebooks
Debugging Kedro nodes became a lot easier with the introduction of the debugging line magic %load_nodefor Jupyter Notebook and Jupyter Lab. The %load_node line magic allows you to load the contents of a node in your project into a series of notebook cells and run them from there.
Learn more about this feature in our docs on debugging a Kedro project within a notebook.
3. Dataset preview
To give users a better understanding of their data, Kedro-Viz now supports dataset previews. Currently, Kedro-Viz shows previews out of the box for the following formats:
TablePreview: For datasets that return tables or DataFrames.
JSONPreview: For datasets returning JSON objects.
PlotlyPreview: For datasets with Plotly charts.
ImagePreview: For datasets returning base64-encoded images.
This is what the preview for a Plotly dataset looks like:
Users can also extend the preview functionality for custom datasets. Learn how to do this and more in our docs about previewing datasets in Kedro-Viz.
4. Powerful configuration management
In Kedro 0.18.5, we introduced the OmegaConfigLoader, replacing the legacy ConfigLoader and TemplatedConfigLoader. This new configuration loader, powered by OmegaConf, brings a range of features that were previously unavailable. Here are two notable enhancements:
Config file merging strategy: with
OmegaConfigLoader, you can specify a custom merging strategy for your config files, moving beyond the default destructive merge. This flexibility allows you to define different strategies for specific configurations directly in your project'ssettings.pyfile. For example:1from kedro.config import OmegaConfigLoader 2 3CONFIG_LOADER_CLASS = OmegaConfigLoader 4 5CONFIG_LOADER_ARGS = { 6 "merge_strategy": { 7 "parameters": "soft", 8 "spark": "destructive", 9 "mlflow": "soft", 10 } 11}In this case, "soft" merging will preserve existing values unless explicitly overridden, while "destructive" merging will completely replace them.
Dynamic value resolvers: instead of hard-coding values in your configuration files,
OmegaConfigLoaderenables dynamic computation of values using OmegaConf’s resolver functionality. Resolvers allow you to define custom logic to calculate values for parameters or catalog entries, or even inject values from external sources—making your configurations more flexible and adaptable.
Learn more about the features of OmegaConfigLoader in the configuration docs.
5. Official Kedro VS Code extension
We’ve added a new project to the Kedro ecosystem: vscode-kedro ! It’s the official Kedro VS Code extension, which integrates Kedro projects with Visual Studio Code, providing features like enhanced code navigation, autocompletion and embedded Kedro-Viz for seamless development. You can download the extension from the Visual Studio Marketplace, which also shows the major features the plugin provides.
6. Customise your project on creation
One of the core value propositions of Kedro is its role as a framework for users to adopt software engineering best practices for data science. When you create a new Kedro project, we set it up to follow the template of a typical, well-structured data science codebase, much like Create React App from the front-end world.
Kedro was created with the view that we should include everything in a new data science project that users need for software best practices. To address the feedback that Kedro was intimidating for new users and too opinionated for advanced users with their own way of doing things, we now enable you to customise what is included in a new project.
You can make a basic, empty, project or create a full-featured spaceflights example that includes every tool we recommend. Or you can put something together customised for your particular needs.
When running kedro new the CLI now offers the option to select the tools you’d like to include in the project.
Learn more about the project creation flow and tools in our docs.
7. Publishing & sharing Kedro-Viz
Sharing pipeline visualisations with your team and stakeholders got a lot easier through the publish and share feature on Kedro-Viz. This feature enables seamless sharing of pipeline visualisations on any static website hosting platform. It's possible to publish Kedro-Viz on platforms like GitHub pages, or deploy it to AWS, Azure, or GCP. Learn how to use this feature in our docs on publishing and sharing Kedro-Viz.
8. Experimental datasets contributions
We've introduced a new contribution model for the kedro-datasets repository to accommodate contributions that are more experimental and may not fully adhere to the usual standards, such as being fully tested. This model allows for the inclusion of datasets that are in the early stages of development or might not meet the criteria for being part of the core Kedro datasets.
Some experimental datasets contributed so far:
langchain.ChatAnthropicDatasetlangchain.ChatCohereDatasetlangchain.ChatOpenAIDatasetlangchain.OpenAIEmbeddingsDatasetnetcdf.NetCDFDatasetrioxarray.GeoTIFFDataset
9. Stateful URLs
Filtering your data pipelines in Kedro-Viz is now more powerful thanks to stateful URLs. After applying filters (such as tags or type), the URL updates to reflect the filtered state. Simply share the URL with others to provide direct access to the same filtered view. This feature allows for quicker collaboration and more focused discussions about specific parts of your pipeline. Learn more about this functionality in our docs on filtering and sharing Kedro-Viz.
10. Slice your pipelines in Kedro-Viz
Pipeline slicing—running a specific subset of nodes within your pipeline—has long been available through the Kedro CLI. Now, we’ve supercharged this popular feature by integrating it directly into Kedro-Viz. With this enhancement, you can visually slice your pipeline, and Kedro-Viz will automatically generate the corresponding CLI command for the selected slice. This makes it easier than ever to execute a subset of your pipeline. Learn more in our short YouTube tutorial about pipeline slicing.
