Things to consider when using Flows for data migration

“Show me a ten-foot wall and I’ll show you an eleven-foot ladder” ― Peter Bevelin

Using Flow for data migration is easy but has limitations you should consider, these limitations can cause problems left the data migration in limbo, whilst waiting for the api limits for each day and Flows waking up and running

When you hit the daily limit for Flows, they stop working until the next day (when limit resets). 

Considerations

  • Data migration can have lots of records and could take many days
  • Flows don’t fail, they queue them up
  • The only way to stop Flows triggering is to delete them or put the Dynamics 365 environment into admin mode.

There are examples where the Flow limits meant the Flows have been in a waiting state for 4 days due to hitting the throttling limit on a daily basis.

Flows are not just on Flow runs by actions inside the flow.

How did we get here with Flows

Microsoft initially had workflows which were hosted by Microsoft and used Async service on the Dynamics 365 server. There were no limits on workflows but from Microsoft’s perspective this isn’t great because these run on Microsoft resources and costs them to host and run them.

Microsoft created Flows, with greater functionality and built on Logic apps. Flows are a scalable, enterprise solution and with connections allow you to create Low code solutions and link systems together.

The benefit to Microsoft is the Flow runs are easily counted and Flows are hosted in Azure which is scalable. Microsoft pushed Dynamics 365 users to use Flows and warn that workflows will be deprecated sometime in the future. Flows allow Microsoft to estimate and control how much it costs to run Flows. Dynamics 365 online is a service, so Microsoft don’t want to have unlimited resource/compute for people.

Flows allow people to scale up and pay for what they use in classic Azure costing process. This initially seems unfair because we were used to no limits but paying for what you use is ultimately fair.

You get taxed one way or another with functionality and it’s not something you can resist. It’s important to align your solutions with Microsoft’s road map. If you resist and continue to use workflows because they are free then in the future you are creating a huge upgrading task.

Flow first

I have a Flow first attitude and workflow should not created on projects going forward. Many Dynamics 365 professionals resist this because they have Workflow skills and might not have created a flow yet. By default we don’t like change and stick with what we know.

Make them do it in a flow, it will be slow to start with but they will soon start to love Flows because they are more powerful

How to cancel flows

There isn’t any way to bulk cancel flows, it’s a one by one event. Flows queue up runs even if they are not enabled!! The only way to stop them is to delete them (and there is no way to bulk delete flows!!)

if you cancel a flow it doesn’t stop the flow running when the api threshold limit is refreshed when you hit a new day.

It’s possible you could be left with 1000’s of instances of flows left in a running state, with you waiting for refreshes of API limits. These could be days!

Deleting flows is only a choice if you have a solution with the flows in and more recommended with an automated release.

Logics apps?

Microsoft have created functionality to run Logic apps on your local machine and some talk that we should all be moving to Logic apps. This combined with an improved editor should make it easier for no code professionals to create logic apps.

I’m not sure about this or if it’s the direction of travel but at the moment Flows are easier because of the current environment connection that means the Flow can work in the environment it’s in and is easily deplorable.

What’s the usual data migration choice

The common tool I see used for data migration on large projects is Kingswaysoft. I haven’t used Scribe online but I believe it has similar functionality, so it depends on the expertise of your team.

Microsoft is moving to count the actions within a flow and step in a flow will be an action which you will have a limit

Dynamics 365 and replicating to an SQL database for reporting

Flow limit

The Flow limit is a topic that will increasingly come up in Dynamics 365 projects, as more projects hit the limit.

My feeling is the limit is a bit low, particularly considering the number of environments you can have in a large project.

Conclusion

Flows are awesome and combined with connections they are powering the no/low code revolution in Dynamics 365/Power Platform projects. Flows and Power Platform are replacing the bespoke .Net applications and Excel spreadsheets that can grow in companies.

Microsoft is starting to be tighter with the Flow limits and charging more in the future. This seems unfair because we are coming from not paying much but in reflection I tend to view the Azure/usage pricing as fair.

Currently the way Flows have some odd ways of work and the turning off functionality can be a nuisance because the runs are queued, waiting for you turn the flows back on. The only way round this is deleting the flow.

Avoid certainty and embrace inquiry

You control the decision, not the outcome #HoskWisdom

There are no right answers, there is only questions and finding out why. There is no certainty, we have questions, ideas and our ability to make good decisions.

The more you think you know, the less you question and the more assumptions affect your decisions.

When you believe you are an expert, you stop listening to other people because you believe there is nothing more to learn. This gives you less information to make decisions with and unable to adapt to changes.

Outcomes

“You cannot make progress without making decisions.” Jim Rohn

A good outcome can come from the wrong decision.

A bad outcome can come from a good decision.

The feedback we get from our decisions comes from the outcome but outcomes are not guaranteed with luck and other variables affecting the outcome. 

We can’t control the outcome but we can control are decision making process. There are occasions where people make the right decision with incorrect logic. This works in the short term because of probability and luck but in the long term the results are based on thinking and quality of your decision making process.

Long term success is based on your critical thinking skills and your ability to solve problems. Environments constantly change and the actions that brought you success yesterday might not work tomorrow.

System thinking says yesterdays solutions are tomorrows problems and this is due to second order effect, unintended outcomes and the environment changes.

You need to make decisions, reflect on the feedback and adjust if you need to.

Don’t be certain, question, clarify and find the why. Being curious helps you uncover information to make better decisions, inquiry helps clarify assumptions.

Certainty in your ideas, stops you changing and adapting to change. It can lead you down the wrong path and your ego can stop you changing your mind.

Situation changes and you have to align with it.

picture from here

Further reading

Why you should listen more than you talk

The Knowledge Project #89 — Maria Konnikova — Less Certainty, More Inquiry

Dynamics 365 October 2020 wave 2 error — Mobile Offline Profiles import: FAILURE: Mobile Offline Profiles import: FAILURE

I was on a project which was manually applying the October 2020 Wave 2 update and after saying this never go wrong, what happens, yep, it goes wrong.

The question is what do you when this happens. In reality you can only raise a Microsoft support ticket.

I raise a P1 with premier support and Microsoft do not reply for 2 and half hours, clearly their idea of a P1 is different to mine.

So I was poking round the system, which is not straight forward because with PowerApps portal there are at least 3 ways to do everything.

So I got to Environments →Dynamics 365 apps

I can see I have two apps with an update available. Of course pressing this button looks like it kicks of an install but nothing seemed to happen.

Image for post

So I look at solution history record in advanced find to see if any solution imports fail. and I find that one of the solution patches from Microsoft has been failing since 15/08/2020. Doesn’t Microsoft check?

Image for post

The error is

Mobile Offline Profiles import: FAILURE: Mobile Offline Profiles import: FAILURE

I found the same error somewhere else

Image for post

The mobile offline profile is if you have a canvas app and you enable it to go offline, which means it can work without internet by downloading data onto the device, e.g. a salesman using a tablet and having the app work offline.

In this case weren’t using offline functionality and the only canvas with offline enabled and the profile was Microsoft’s Field Service app.

I have had problems with the mobile offline profile before

Dynamics 365 — mobile offline profile error

It would surprise me if this is the problem but I didn’t really have any method to investigate. The way the patches are applied I can’t see the import logs to identify where the errors are.

What’s surprising is Microsoft best practices is not to use patches and yet they use them. They are pushing regular update of patches and some of them aren’t working and no one is informed or does anything about it!

Image for post

I will wait to see what the resolution is to the problem but it was frustrating because I couldn’t do much to investigate or fix it. The problem is with Microsoft updates and they will have to investigate.

Why you should listen more than you talk

“Never give reasons for what you think or do until you must. Maybe after a while, a better reason will pop into your head.” General William T. Sherman

When you are trying to impress people with words, the more you say, the more common you appear, and the less in control. Even if you are saying something banal, it will seem original if you make it vague, open-ended, and sphinxlike. Powerful people impress and intimidate by saying less. Robert Greene

The more someone talks the more you learn, the more we talk the more information we give away. Despite this, our inclination is to talk more than we listen. Listening lets us learn more about a situation, gather more information. Talking has the opposite effect, you give more information away.

Shut up and listen

The more people talk, the more points they make, the increased chance of contradicting themselves. The more points you make, the more ammunition you give to people to disagree with you, let them wonder what you think and give yourself room to change your mind.

We waste time by talking about things which aren’t important, focusing on the wrong areas, instead, keep quiet.

You learn by listening

When you speak you don’t learn. When you listen, you might hear extra information, a new idea, feelings or a new perspective. When you listen you learn.

What is the goal?

  • What is the goal of this discussion?
  • What is the motivation of the people?
  • What is being discussed?
  • What problem is being solved
  • What does the other person think?
  • Why do they think that?

When you listen it helps you understand what people think, why they think something and how they feel. Yet before they have time to speak, we are telling them about our ideas on the topic.

You produce better ideas if you explore problems, understand different perspectives, and gather more information before forming ideas. The earlier you speak the less information you have used to come to your conclusion and the great the chance of misunderstanding the situation.

Better solutions need you to listen and learn, if speed is the goal then talking can help you push forward a conversation and decision. The earlier you speak the less informed your answer will be.

Perspectives

There are no right or wrong solutions, each solution has strengths and weaknesses. Understanding different solutions from different perspectives allows you to create solutions by combining different ideas.

Silence makes people nervous

People don’t like silence and they will talk to fill it. The more people talk, you the more you learn about them, their thoughts and their ideas.

Less is more

”The human tongue is a beast that few can master. It strains constantly to break out of its cage, and if it is not tamed, it will run wild and cause you grief.” Robert Greene

Speaking less reduces your chances of saying something stupid or saying things which are incorrect. speaking more, increases the chances of you saying something stupid or something you will regret.

Time to think

The less you speak, the more time you have to think. You can’t talk and think, you can’t listen and think. When you listen you gather information, when the speaker stops, you think about the new data and how it fits with the existing data you had.

Ask questions

When you ask questions, you get answers. If you disagree, you can ask a question and highlights the weakness in their ideas. Don’t tell people they are wrong, ask questions and help them change their own minds.

Changing minds

Time arguing is time wasted and unsuccessful. So do less of it and focus on being positive. Don’t waste time on negativity.

Getting requirements

A mistake people make when getting requirements is stop listening and move to creating the solution. They stopped listening they didn’t create the solution that user needed because they didn’t understand the requirement.

Conclusion

Listen more than you speak, focus on acquiring information rather than sharing it. You gain little less by talking and more from listening. 

The more you talk, the more you give away. Listen, learn and follow Robert Greene’s 4th Law — always say less than necessary