When a Patch to Microsoft Dynamics 365 can break your instance

 

“When you come to the end of your rope, tie a knot and hang on.” — Franklin D. Roosevelt

We had problems importing solutions in a sandbox instances for Microsoft Dynamics 365. It was because different instances having different versions of Microsoft Dynamics 365. This article looks at the potential problems of patches.

Upgrading

We have been upgrading customisations to make them Microsoft Dynamics 365 version 10 compliant and removing the customisations which are depreciated in Microsoft Dynamics 365 version 9.

You can find a list of the important changes and what’s depreciated here:

https://docs.microsoft.com/en-us/dynamics365/get-started/whats-new/customer-engagement/important-changes-coming

There can be confusion about what you need to update with new versions. Microsoft supports the last 2 releases: Microsoft Dynamics 365 version 8 and Microsoft Dynamics 365 version 9. Customisations are depreciated in Microsoft Dynamics 365 version 9; they will still work in the current version, but it means you can’t use them in Microsoft Dynamics 365 version 10.

An example if the client api is depreciated

Xrm.Page

Now need to get the FormContext

executionContext.getFormContext();

so before you would have done this

var firstName = Xrm.Page.getAttribute(“hosk”).getValue();

now you need to do this

var formContext = executionContext.getFormContext();

var firstName = formContext.getAttribute(“hosk”).getValue();

You can make the change now but you only HAVE to make this change when you are using Microsoft Dynamics 365 version 10.

Patches

The Capgemini Dynamics team store customisations in source control, this allows us to create solutions and deploy them into new environments (we can create an environment and deploy all customisations with a click of a button). We can see the changes in Microsoft Dynamics 365 customisations.

We compare the xml changes of forms, views, JavaScript and all customisations. This week we noticed in the last patch Microsoft added a new section and JavaScript to the contact entity.

This raised the following questions:

1. Should this change happen in a patch release?

2. With Microsoft’s new direction of making all customisations as solutions, wouldn’t this functionality be optional?

3. What functionality could Icebreakers be? Would Microsoft suggest Icebreakers or you store them for each contact?

4. Patches are not meant to break functionality

Problems

A new section added to the form which could confuse testers and users, but it breaks the deployment because it added a dependency to new internal solution added.

We got an error about missing dependency to “Icebreaker ” solution and msdyn_talkingpointsloader.js web resource during deployment testing on a separate tenant. The sandbox instances were upgraded with the latest patch, while extracting the latest changes, the section changes with some web resources and controls, adding dependency to “Icebreaker” solution.

Dev instance is version

Test instance is version

Between 9.0.2.54–9.0.2.189 (DB version) this solution and dependency was added, making it difficult to use multiple tenants because Microsoft controls when patches are installed. You can get into a situation where some of your instances have had a patch applied whilst you are waiting for Microsoft to apply patches for others.

When we put code into source control and include Microsoft’s changes, we then cannot install these solutions into the instances which have not had the patch applied.

Here is the Unhandled Exception:

System.ServiceModel.FaultException`1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=9.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]: The action was failed after 0 times of retry. InnerException is: Microsoft.Crm.Tools.ImportExportPublish.ImportSolutionException: Solution manifest import: FAILURE — -> Microsoft.Crm.Tools.ImportExportPublish.ImportSolutionException: Solution manifest import: FAILURE — -> Microsoft.Crm.CrmException: The following solution cannot be imported: feature_Customisations. Some dependencies are missing. The missing dependencies are : ><Required key=”1510″ type=”66″ schemaName=”MscrmControls.TalkingPointsControl.TalkingPoints” displayName=”MscrmControls.TalkingPointsControl.TalkingPoints” solution=”Icebreakers (9.0.1.7089)” /><Dependent key=”125″ type=”60″ displayName=”Contact” parentDisplayName=”Person” id=”{1fed44d1-ae68–4a41-bd2b-f13acac4acfa}” /></MissingDependency><MissingDependency><Required key=”1536″ type=”61″ schemaName=”msdyn_talkingpointsloader.js” displayName=”msdyn_talkingpointsloader.js” solution=”Icebreakers (9.0.1.7089)” /><Dependent key=”125″ type=”60″ displayName=”Contact” parentDisplayName=”Person” id=”{1fed44d1-ae68–4a41-bd2b-f13acac4acfa}” /></MissingDependency></MissingDependencies>

Solutions

You only have a few days’ notice and you have no ability to control when patches are applied. Some Dynamics instances were on one version and the others instances waiting for the patch to be applied. This can create solutions which can’t be imported into some of your instances. You must monitor when patches will be applied and check if all your instances have had the patch applied.

The documentation surrounding patches is sparse and there can be significant changes. There is a risk of a patch breaking customisations in production — with a small time to test, how do you mitigate this problem? If Microsoft gave a detailed list of the changes it would help to risk assess the patch.

When I worked on projects using Microsoft Dynamics CRM on premise, the best practice was to never take the latest patch but install the patch before that. This way you could buffer yourself from Microsoft breaking Microsoft Dynamics. I wrote a blog post on it Should you keep up with Microsoft Dynamics CRM release cycle?

Microsoft could implement the latest patch to be a beta version and make it operational or move to a monthly release schedule.

In the past when patches broke production, the infamous patch of Microsoft Dynamics CRM 2011 patch 12 — where Microsoft changed how JavaScript worked and broke many instances. I discuss it CRM 2011 — Things learnt when reviewing Javascript code on form loads

Conclusion

Most of the times patches come and go, fix bugs and no one notices. It’s good that Microsoft regularly fixes bugs — that benefits its users. If Microsoft could improve its documentation, give more warning and make significant changes in more significant patch numbers this would resolve many of the problems. This is the online service world we live in and it’s a more exciting way of living.

Advertisement

How we try to speed up IT projects and why it doesn’t work

 

Code not in production has no value and you don’t know if it will work, get it live in the shortest time possible #HoskCodeWisdom

How does a project get to be a year behind schedule? One day at a time. Fred Brooks

Code not being used by users is worth nothing, you to get a return on investment of creating the code only when it is being used in production by users.

People get lost in the fog of a project and forget its purpose is to create software which helps a business achieves its goals and do the activities which makes the business money.

The goal of IT projects is to create software to help a company deliver its business goals. To help the employees finish tasks and the company to be profitable.

No one cares about technology, employees want to keep their job and the business to be profitable. Technology is an aid to doing work easier, quicker or automating it.

To get a return on this investment of creating software, you need employees using it.

 

Until the code is in production, it has no value to the business #HoskCodeWisdom

Releasing quickly

I tweeted the quote and it generated a some negative answers on LinkedIn, much of which Some of the blame is mine for for the poorly worded statement

Here is the link

This generated comments

“I hope you mean after all quality gates have been passed. If not it makes no difference if you code directly in production. Since it was not tested anyhow.”

“#hosklowqualitycode”

“If that’s your attitude, I would stay well clear of you delivering any code-heavy solution if your priority is to rush stuff into production. This sounds like something written by someone to whom quality, and quality assurance is very much a secondary consideration. Stating that you think that code should, and can only be tested in production implies that you have place a very low premium on the testing process.

“Sounds like recipe for disaster; imagine if such code was behind human nuclear launch systems. If so, recipe for human extinction! if that is your culture, I expect they will also be extinct in due course. Code not in production worthless,? have you not heard of IP?”

When I wrote the comment I focused on speeding up deploying code into production using DevOps, rather than using shortcuts.

The assumption from some readers was a faster release schedule means reducing quality and not testing.  This is the mindset used before DevOps and continuous integration, continuous deployment.
Software engineers should not be spending their time doing repetitive tasks, which can be automated.

If your deployments are manual and you don’t consider automation then it’s difficult to speed up the deployment process with using short cuts and creating technical debt for the project.

Traditional thinking of speeding up projects

The traditional method of speeding up projects with manual deployments is taking shortcuts or working more (weekends, evenings) or faster.

I have written about how rushed projects leads to lower quality  

and What’s slowing down your CRM development? for other thoughts on development/release cycle and project quality.

Common methods to speed up projects

• Add more people
• Work weekends
• Reduce quality
• Reduce testing
• Reduce scope

The only one of these methods I would agree with is reducing scope and this is the choice customers don’t like choosing.  One benefit of Scrum is the simple rules regarding scope e.g. if you add a story to the sprint, you must remove a story but often people want to squeeze in one more story but Scrum doesn’t allow that.

Add more people

Adding manpower to a late software project makes it later. Fred Brooks

The more people you add to a project, the more complexity you add.  Communication and collaboration becomes harder and the project needs to spend more time communication to avoid creating conflicting solutions, making incorrect assumptions and creating a solution which works.

I tried to explain the problems of adding more developers is to imagine 9 people all trying to write a chapter of a book at once, each person writes 5 pages.  All 9 people need to know the characters, the plot, what’s happening now, what’s happening in the future, writing standards, where to store it.

The chances of the chapter being written 1st time is tiny,  it will take many iterations before you create a coherent story.  This is like a team of developers creating one solution, communicating, checking code works with the existing code, code works as specified by the requirements.
Adding more developers increases the technical debt because they work on seperate parts of the code base.  Good developers will do it properly and refactor as they go but good developers are hard to find.

Problems that come with adding more people are also known as Brooks’s law from the author of The Mythical Man-Month: Essays on Software Engineering

Adding manpower to a late software project makes it later. Fred Brooks 

Brooks explains the problems in seeing people as interchangeable better than I can

“Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable. Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them… This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming” Fred Brooks

e.g. Software development is difficult and a project needs people to understand all the intricacies of this project (Business knowledge, people, existing technical solution, tools, etc)

Brooks’s Law focuses on projects who are already late but the main points

• It takes some time for the people added to a project to become productive.
• Communication overhead of adding more people
• Limited divisibility of tasks

The bigger the project the harder and longer it takes people to become productive.   On Boarding is underestimated.  When starting a project you need to learn

• Business knowledge
• Existing technical solution
• People on the project
• The way the project works, it’s flow
• Tools and software used on the project

The more people the harder you need to work communicating with them all, the more meetings that pop up!  You can end up slowly down the project by adding more people, the opposite of what is intended by adding additional people.

The last point on limited divisibility of tasks refers to you can’t always split up tasks for multiple people to work on simultaneously.  Summarised by this quote

“nine women can’t make a baby in one month”. Fred Brooks

Work weekends

You can do more work, if you work more days and hours in the same time, by working weekends, the elapsed project days stays the same but the team can deliver more.

This can work on a short-term basis e.g. but it should be a last resort.  Working weekends is a sign of problems, the development team have too much work to do.  It burns out the developers, giving them no time to rest and recharge.

Longer hours results in developers working harder but not more getting done because IT projects are about quality development.  Development needs thinking, creativity and experimentation.

When development teams work harder they lose their spark, churn out code of lower quality and contains more bugs.  Private lives suffer, weakening their relationships with friends and family.

Reduce quality

Faster development leads to lower quality.  Taking shortcuts, cutting corners, delivering to lower standards seems to get the project ahead but it’s an illusion.  The code will be more complex, you have built up technical debt and the debt will be paid later in the project.

The project will slow down when existing code needs changing or extended, the lower quality (highly decoupled, brittle code) will be harder to change and take longer.

Lowering quality gets you ahead now, only to fall behind later

Reduce testing

Unit testing, automated testing and a tester gives the developer a short feedback loop.  Short feedback loops allow you to test your code and find problems and bugs, you fix them whilst deep into the code and an expert.  The further away (longer) your feedback loop is the slower it is to fix bugs.  It takes longer for the bug to get back to the developer and they know less about the code, resulting in longer to fix and retest.

Development is about creating quality, fix code nearest its creation to be most efficient.

Reduce scope

You can speed up a project by reducing the scope.  Less software to create, equals less time to develop and test.

This is effective but unpopular because no one wants to give up functionality.  I rarely see this implemented.

I’m a fan of delivering simplified functionality to get it into production and get feedback.  Often this is good enough.  Simplified functionality is rarely agreed in principle but can gain momentum when seen in action.

Summary

Working functionality is the measure of all projects (assuming its by the users), the smaller the release cycle, the quicker we generate value.

Large IT projects are difficult, the bigger the project the harder it is.  I have read statistics that projects which go longer than a year have a 50 percent failure rate (I can’t remember where I read that state).  This post covers The difficulties of scaled agile projects.

The traditional measures of speeding up projects are adding more people, working harder or reducing quality (by doing less testing).  Scaling the team should only be done when the existing team is working well.  Reducing quality never gets you a head, it just saves the problems for later.

Reducing scope is a realistic approach, maybe not removing scope entirely but delivering a simpler version to begin with, let it be used and make changes from feedback.  You can then deliver smaller updates and everyone feels good.

Most attempts at speeding up projects have been unsuccessful, it often seems people get to a situation where they must hit a deadline but without anyone really thinking why or how this particular date was decided.

The goal of a project is to create a solution which works, it’s better to create a quality solution which takes a bit longer than rush something which doesn’t work.  Production issues cause panic, it’s better to fix these when the stakes aren’t as high.

A great book on the problems of IT Projects (mostly people related)

• Peopleware: Productive Projects and Teams
• Managing Humans: Biting and Humorous Tales of a Software Engineering Manager
• The Mythical Man-Month: Essays on Software Engineering

I recommend the book for anyone working on IT projects, also developers should read books

Where is the better way

This blog post got to big and I got carried away so I have split it up into two posts. in the next post I will discuss DevOps.  I got engrossed in Brooks law and thinking about projects I have worked on and the various attempts deployed to speed up them up.