Landing this month in #Microsoft #Fabric #DataFactory pipelines ... Edit your JSON code behind the canvas designer!

#MicrosoftCommunity asked and we listened! Thank you as always for the feedback.

Invoking data integration ETL pipelines when a file arrives or deletes is a common pattern when you cannot rely on a wall-clock scheduling mechanism.

Watch as Mark @KromerBigData walks through how to do this in #Microsoft #Fabric #DataFactory and more!

youtu.be/kLko6bWy9sw

Are you an experienced #Azure #DataFactory pipeline developer and want to hear 5 tips about transitioning from ADF to #Microsoft #Fabric #DataFactory!

Mark @kromerbigdata has you covered:

5 Tips for ADF pipeline developers moving to Fabric

youtube.com/watch?v=PF6A-f…

In this new #Microsoft #Fabric blog, @KromerBigData continues his blog post series on #DataFactory data pipelines in Fabric by demonstrating several different methods and good practices to automate and execute your Fabric DW queries and commands.

blog.fabric.microsoft.com/en-us/blog/aut…

In this new video, @KromerBigData shows you how to make a new Dataflow in #MicrosoftFabric #DataFactory that does exactly the same transformation job a #MappingDataFlow in #Azure #DataFactory.

youtu.be/4-1Tu2UoEns

🚨🚨

We've just updated the Flatten transformation so that you can now easily unroll multiple arrays from a single Flatten transformation step!

Now available in both #Azure #DataFactory and @Azure_Synapse data flows.

techcommunity.microsoft.com/t5/azure-data-…

Hey #Azure #DataFactory fans!

@KromerBigData demonstrates the latest enhancements we've made to CDC in ADF with micro-batch change data capture and schema drift ... all in dark mode!

youtu.be/OF6PeYMWqKg

We've just released the public preview of the pipeline return value feature!

Pipeline creators can now customize the output value of a pipeline and consume that value in the caller.

This makes the common child-parent pipeline pattern more more powerful!

techcommunity.microsoft.com/t5/blogs/blogw…

@KromerBigData demonstrates the latest Change Data Capture features in #Azure #DataFactory.

Build a CDC process in minutes without a pipeline, data flow, or trigger!

youtu.be/YTxANiAPAKs

@JohannesVink You need to hover over the port until you see the finger pointer, then you can add more activities to that port.

@ImAbhishekTomar @AzureSupport @AzureCosmosDB @GauravMalhot12 Make sure the dataset type "Inline" is selected from the sink.

@ImAbhishekTomar @AzureSupport @AzureCosmosDB @GauravMalhot12 The settings you can choose for the container are properties on the sink settings tab:

@ImAbhishekTomar @AzureSupport @AzureCosmosDB @GauravMalhot12 Yes, you can do this in a data flow. In the Sink, set the sink type to an "inline" Cosmos DB dataset. On the sink settings tab, you can parameterize the name of your container to create on the fly. Lastly, be sure to select "Recreate container" as your Container Action.

@NowinskiK @KromerBigData You may need to refetch the source definition by invalidating the data preview cache. Go to the source transformation and click Refresh > Refetch from sources

@KromerBigData walks through several good practices, tips, and tricks when debugging data flows in #Azure #DataFactory and @Azure_Synapse

Learn how to use & explore data samples for the best debugging experience & right-size the debug environment.

youtube.com/watch?v=pVvHsk…

📢ICYMI: Built-in support for native SQL change data capture is available for #Azure #DataFactory & @Azure_Synapse!

Check out this demo by @KromerBigData to see it in action: youtu.be/Qu9u69pkC5o

Don't forget to join our livestream TODAY at 11:30am PST / 2:30pm EST to see what's new in #Azure #DataFactory!

youtube.com/watch?v=Bh_VA8…

#WhatsNew #AzDataFactory

Join our livestream on Sept 28 at 9am PST / 12pm EST to hear more about SAP updates in #Azure #DataFactory!

Link to livestream here: youtu.be/8w7kD2EbbYI

#AzureDataFactory #SAP

🚨 Exciting new UI updates coming to #Azure #DataFactory and @Azure_Synapse pipelines!

@KromerBigData details these upcoming changes previously available in the preview experimental view. These will become the default in the pipeline designer!

techcommunity.microsoft.com/t5/azure-data-…

Hey #Azure #DataFactory and @Azure_Synapse fans!!!

We're going live soon with our monthly update!
Join our livestream at 4:00pm PST / 7:00pm EST:

youtu.be/EOVVt4qYvZI

Hey #Azure #DataFactory and @Azure_Synapse fans!!!

We're going live soon with our monthly update!
Join our livestream at 10:30am PST / 1:30pm EST: youtu.be/Ay3tsJe_vMM

Did you notice the new "Errors" tab on the #MappingDataFlows sink in #Azure #DataFactory and @Azure_Synapse ?

You can now log rows that fail your own custom assertions!

Watch this intro video as Mark (@KromerBigData) explains this new feature:

youtu.be/VFRx0wjlA4s

@AzureDBPostgres @halilozanakgul @Azure Also brand new is simple incremental delta data loading from Azure DB for PostgreSQL in #Azure #DataFactory and @Azure_Synapse

Automatic incremental source data loading now available natively in #Azure #DataFactory and @Azure_Synapse for Synapse SQL, Azure DB for PostgreSQL and MySQL.

REMINDER ⏰⏰

Tomorrow, April 12 at 12 PM PST, see the latest updates and demos of #Azure #DataFactory from @NoelleLi3 and @KromerBigData YoutTube Live Stream

youtu.be/MkgBxFyYwhQ

@KromerBigData walks you through an example of transforming Avro data from Event Hubs using Parse and Flatten transformations in #mappingdataflows

#Azure #DataFactory @Azure_Synapse

youtu.be/F2x7Eg-635o

Hey #Azure #DataFactory and @Azure_Synapse fans!!

The March product updates live stream has been scheduled for April 12 at 12 PM PST.

youtu.be/MkgBxFyYwhQ

@MapEngArch Here is an example of injecting an event into Event Grid from an ADF pipeline Web Activity. See if it helps ... youtube.com/watch?v=FpKrBL…

Hey #Azure #DataFactory fans!

The ADF team is about to go live over on YouTube for a 30-min session of demos on new features and live Q&A!

youtu.be/r22nthp-f4g

@KromerBigData walks through an example pipeline in #Azure #DataFactory using new features recently released in ADF:

Asserts
New schema drift function
SQL Script activity

youtube.com/watch?v=xukIkh…

#Azure #DataFactory No-code Data Preparation in a Modern Data Warehouse with @PowerQuery
youtube.com/watch?v=zHR1W-… @CanCloudSummit

Use #Azure #DataFactory Flowlets in ADF or @Azure_Synapse to build composable reusable logic units of dataflow code and add to pipelines as your own transformation blocks!

Great video by @MAHEERBASHA !

youtube.com/watch?v=zYB3bx…

Great example here of how to use a Flowlet in #Azure #DataFactory and @Azure_Synapse #mappingdataflows for make your logic reusable across data flows and pipelines.

youtube.com/watch?v=0omf1v…

🚨🚨🚨

New!! Just released public preview of source & sink connector for #mappingdataflows in #Azure #DataFactory and @Azure_Synapse:

Dynamics 365

#Azure #DataFactory Live Stream by Data Platform Central: Multi-Streamed Data Wrangling Transformations​ @PowerQuery

youtube.com/watch?v=0-uEbN…

Very nice demo video here on how easy it is to set-up the @AzureCosmosDB change feed for #Azure #DataFactory to utilize our new CDC feature.

This new change capture feature is also available in @Azure_Synapse pipelines and data flows.

youtube.com/watch?v=g2rJpK…

Thank you to the #Microsoft #Azure community & #DataFactory customers in helping us make ADF the best product for data engineers. From the Microsoft Azure Data Factory product team ... thank you for your continued support & making 2021 so special!

youtube.com/watch?v=EEbki9…

Nice easy to follow quick demo of the Rank transformation in #Azure #DataFactory and @Azure_Synapse #MappingDataFlows

youtube.com/watch?v=-tuYJO…

Nice walk-thru demo here of Flowlets for building data flows from reusable components for #Azure #DataFactory and @Azure_Synapse #mappingdataflows

youtube.com/watch?v=VWjhbD…

@KromerBigData @Azure_Synapse

Quick Spark cluster re-use for Azure IR has landed in @Azure_Synapse for #mappingdataflows.

No need to explicitly set "quick re-use", all TTLs in Synapse will start sequential data flows in < 3 seconds.

This will become default behavior in #Azure #DataFactory soon as well.

@KromerBigData demonstrates how to make your data transformation expressions dynamic with pipeline parameters in this #Azure #DataFactory demonstration of data quality rules in an external configuration file.

youtube.com/watch?v=q7W6J-…

@KromerBigData demonstrates how to delete missing source rows from your target database using #mappingdataflows in #Azure #DataFactory @Azure_Synapse by using Exists and Alter Row transformations.

youtu.be/9i7qf1vczUw

The #Azure #DataFactory Power Query activity now allows you to map your transformed columns to different target columns in your database destinations. docs.microsoft.com/en-us/azure/da…

@PowerQuery

#Azure #DataFactory and @Azure_Synapse data transformation using pipelines and #mappingdataflows

@WintellectNOW

youtube.com/watch?v=oPj7T9…

Here are some helpful links to learn how to get current row context in #mappingdataflows and use that to compare column values against current and previous rows.

#Azure #DataFactory @Azure_Synapse

docs.microsoft.com/en-us/azure/da…

youtube.com/watch?v=jqt1gm…

You can now edit existing Azure IR data flow settings, without needing to create a new IR when you change your compute type and core counts in #Azure #DataFactory and @Azure_Synapse #mappingdataflows

#Azure #DataFactory No-code Data Preparation in a Modern Data Warehouse by @ThimanthaV

@PowerQuery

youtube.com/watch?v=92FO_B…

Activity execution time without and then with TTL set in Azure IR for data flows in @Azure_Synapse

New in @Azure_Synapse data flows ... Support for TTL in Azure Integration Runtime to make your data flow execute in seconds, not minutes, just like in #Azure #DataFactory!

Watch WafaStudies demonstrate how to use the #Azure #DataFactory @Azure_Synapse Parse transformation to parse embedded JSON out of SQL data

youtube.com/watch?v=42OAjO…

Split single data row into multiple rows using #MappingDataFlows in #Azure #DataFactory and @Azure_Synapse

youtube.com/watch?v=lAHu79…

@cloud_maha @KromerBigData Click on Define Default Format in the Source Projection panel and enter values other than 0 for Boolean True and Boolean False so that ADF's type inference can tell the difference.

Watch as @KromerBigData demonstrates data profiling and data cleansing in #Azure @Azure_Synapse with data flows and pipelines

youtu.be/et1w9C8SRq4

We love learning from the #Azure data community!

Tell us how you are using data integration on Azure by sharing your #DataIntegrationBestPractice with Azure #DataFactory and @Azure_Synapse pipelines and data flows.

aka.ms/di/bestpractice

You can now build complex field mappings to your #powerquery sink using #Azure #DataFactory data wrangling! Please note that the sink is now configured in the pipeline in the Power Query (Preview) activity to accommodate this update.

docs.microsoft.com/en-us/azure/da…

New this week in Data Flows!! #Azure #DataFactory @Azure_Synapse

You can now set custom umask properties on ADLS Gen2 sinks.

Custom event trigger can enable event-driven data pipeline triggers in #Azure #DataFactory and @Azure_Synapse

ADF Custom Events for file watcher and pipeline chaining youtu.be/FpKrBLeqdj4

@gunjandadhich You can apply SHA hashing on your data columns in ADF using data flows: youtube.com/watch?v=OFd4Le…

Brand new!! The data flow cache sink allows you to pass back results of your data flow calculations to the pipeline without ever writing any data out!

Watch as @KromerBigData walks through an example

#Azure #DataFactory @Azure_Synapse
#mappingdataflows

youtu.be/r1m3Ya14qpE

ETL in #MicrosoftAzure Made Easy with #DataFactory #MappingDataFlows by @mrpaulandrew and @SQLBits

@Azure_Synapse

youtu.be/KQLE8zywnTE

Data Flows in #Azure #DataFactory and @Azure_Synapse enable data engineers to transform complex data structures like maps, hierarchical types, arrays ... all in a graphical low-code ETL environment.

Watch as @KromerBigData shows you how it's done!

youtu.be/Wk0C76wnSDE

**Attention #Azure #DataFactory users**

Next week we will finally remove the Parameterization template link from Manage hub. Hooray!

You've been redirect to "ARM template" for awhile now, but wanted to give a heads-up that your UI will look a little different next week.

Watch @KromerBigData demonstrate our new public preview option for data flows in the Azure Integration Runtime that executes data flows sequentially in a pipeline in seconds!

youtu.be/mxzsOZX6WVY

New UI update coming to #Azure #DataFactory and #SynapseAnalytics data flows!

You can now see the compute capacity for the Integration Runtime associated with your debug session.

Note that this is moving from a modal pop-up box to a fly-in panel.

@greglow @KromerBigData Do you mean the query property in the Source like this?

#Azure #DataFactory @Azure_Synapse

Product roundtable w/ADF product team recorded live stream discussion.

Latest product updates post-Microsoft Ignite 2021 for ADF & Synapse pipelines, data flows, Azure Purview integration.

youtu.be/lOjjHXvYKMs

Join the #Microsoft #Azure #DataFactory product team for a set of Lightning talks and Product Roundtable. We greatly value opportunities to engage with the community to hear how we can work together to build better products together. @Azure_Synapse

meetup.com/Microsoft-Reac…

New category coming to the data flows transformation toolbox this week! You'll now find the Parse and Flatten transformation under "Formatters".

#azure #datafactory @Azure_Synapse

Sample scenario using Data Flows in #Azure #DataFactory and @Azure_Synapse to prepare data for AutoML models:

* Partition large data to parquet files before training models
* Split training datasets and test datasets
* Remove unqualified data

docs.microsoft.com/en-us/azure/da…

Intro video to using the new data flow Parse transformation in ADF and @Azure_Synapse.

Extract / parse embedded JSON and other data formats inside string columns.

youtu.be/r7O7AJcuqoY

#Azure #DataFactory is expanding the #PowerQuery activity Sink options.

We're moving Sink options from the fly-in panel to the bottom panel.

In the coming weeks, we'll open up more Data Flow scale-out and mapping options to give you even more power to execute PQ/M at scale!

@dgcaron @ErwindeKreuk @narainabhishek @jasonhorner @MMarie @NowinskiK @AndyLeonard @mrpaulandrew @sjoerdborneman The previously required manual step of clicking publish in the ADF UI has been relaxed in the new CI/CD workflow enabled by the npm package:

Here is an #Azure #DataFactory @PowerQuery ADF Pipeline Example where @KromerBigData shows how to ingest data from any ADF connector into a storage account for data prep with PQ and slowly changing dimension handling with ADF data flows.

youtu.be/3FJgd2OD5GI

@MyMemoryLeaks @KromerBigData We do not have a library to auto-gen the data flow script. But you can build the text script interactively in the ADF browser UI. The script generation is visible using the Script button and includes a button to prep the script as a string to send to the create data flow API.

#88. Azure Data Factory - Process Fixed length/width file in ADF in 2 steps youtu.be/6HawgLhYHuo

Real-World Data Movement and Orchestration Patterns using #Azure #DataFactory youtu.be/DhcPjWeA8K4

by @jasonhorner

Deep Dive: Enterprise Data Integration Simplified with Azure Data Factory and performance of ETL in the cloud using Data Flows in ADF and #AzureSynapse analytics.

With @AnalyticAnna @weehyong @KromerBigData

youtu.be/NaxDm7IQ0lE

A great, big Thank You and 2020 send-off from the #Microsoft #Azure #DataFactory team!

Thank you to our customers and the wonderful Azure and #SQLServer communities.

Wishing everyone a happy and healthy 2021!

youtu.be/V5MHW-E2Voc

Here are some tips from @KromerBigData on ETL in #Azure Data Lakes with Data Flows youtu.be/Samj4b_ZSrY for #DataFactory and #SynapseAnalytics

Want to hide merged changes in a production environment? @big_data_dan shows how use global parameters and the if activity to add feature flags to do exposure control in ADF!

youtu.be/WvDRM4cEk2A

#Azure #DataFactory #CICD

Cache Sink and Cached lookup in Mapping Data Flow in Azure Data Factory youtu.be/9d39Bw2uFLw

#Azure #DataFactory #Synapse

ADF Adds Cached Lookups to Data Flows #mappingdataflows in ADF and Synapse studio!

youtu.be/HqCn42FaRJs

Quick intro to the new Rank transformation for ranking across entire datasets.

Rank data in your data lake with ADF and Synapse Data Flows youtu.be/6XvgkbMtws0

#Azure #DataFactory #Synapse #mappingdataflows

@js_0505 @AzureSupport In Copy Activity Source setting

Easily add generic row deduplication and NULL checks in ADF and Synapse using #Azure Data Flows with #mappingdataflow snippets.

youtu.be/K785tr7NC3o

Data flow improvement landing in ADF this week: Toggle whether a 'No files found' error if no files match from a source. Available in all file-based sources in mapping data flow! If true, no files = 0 rows read

#Azure #DataFactory #MappingDataFlow

Join the #Azure #DataFactory product team on Sept 15 at 11:30 AM PST for a 30-minute Live Stream briefing of exciting new features and updates coming to the ADF Data Flows expression builder UX.

@KromerBigData @big_data_dan

youtu.be/dBXF-8tGOD4

#Azure #DataFactory Hierarchical Data Transformations youtu.be/oAEh21NFgWQ

Work with hierarchical data in Azure Data Factory with Data Flows for data transformation of complex data types at cloud-scale.

#Azure #DataFactory #mappingdataflows with @SnowflakeDB youtu.be/WrF8Qmc7MEU

#Azure #DataFactory with Data Flows for code-free data transformation at scale in the cloud by @SQLPlayer.

youtu.be/uUfygAtSfcg

#mappingdataflows

#Azure #DataFactory Tour with @grrl_geek Jes Schultz youtu.be/ThUzLcdb3EM