Dynamics 365 — how solution layering can help you resolve solution updating problems

A problem clearly stated is a problem half solved. Dorothea Brande

I was importing a managed solution into my test environment, only to find it wasn’t updating the Canvas App. There can be many reasons for solutions not updating and this post will discuss the cause and the solution.

Environment setup

Below are the environments we use.

Dev → Dev Master → QA/UAT/Training

Developers make changes in the Dev environment and creates a patch, deploys to Dev master. Dev Master is the environment is used to create our managed solution, which is imported into the non development environments. It’s managed because no one should change customisations outside of the Dev environment, this helps to control the environments and keep them all in Sync.

Once your environments are out of sync your testing becomes unreliable because you can’t be sure what customisations are triggering and working to in the whole solution.

The problem

A model driven app was updated, adding more views to an entity but when deployed to the QA environment the new views were not showing.

  • Dev + Dev Master — 10 views
  • QA — 3 views

What was causing QA to only show 3 views?

I wanted to confirm the changes had been checked into source control and the AppModule.xml had been updated.

The solution packager breaks down all the customisations in your solution into xml files. This is great for checking into source control and managing your customisations. Below are the folders it creates.

In the AppModules folder, I found these

I found the entity I had changed (no it’s not really called HoskEntity)

<AppModuleComponent type="1" schemaName="HoskEntity" /><AppModuleComponent type="26" id="{aba4dde6-7a10-ea11-a811-000d3a4ab96a}" />
<AppModuleComponent type="26" id="{b174159f-04b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{2fe72247-43b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{b3589b99-42b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{001594bd-43b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{7bd435d8-43b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{3a2bdec8-7a10-ea11-a811-000d3a4ab96a}" />
<AppModuleComponent type="26" id="{19a41b6a-43b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{1670e091-43b9-e911-a82e-000d3a47cb1d}" />
<AppModuleComponent type="26" id="{ffa5f67d-43b9-e911-a82e-000d3a47cb1d}" />

If you can’t read Dynamics xml, this page AppModuleComponent helps

  • Type 1 = Entities
  • Type 26 = Views

My entity has 10 views, which is what I expected and confirmed the problem isn’t here. Bug fixing isn’t about finding the problem straight away, every assumption you clarify takes you closer to the cause of the problem.

QA Environment

In the QA environment to the model driven app and pressed the solution layering button. This is a useful tool enabling you to see what customisations are layered on top of each other. Different customisations in different orders can produce different results.

Solution layering button has two other useful features.

you can click on the layer and see the customisations

There is a button which allow you to remove those customisations (if you click the three dots).

Read these articles to learn more

The customisations on the default layer, show the xml of the AppModule. I searched for the entity I had changed and found it only had three views.

<AppModuleComponent type=”1" schemaName=”HoskEntity” />
<AppModuleComponent type=”26" id=”{b174159f-04b9-e911-a82e-000d3a47cb1d}” />
<AppModuleComponent type=”26" id=”{2fe72247–43b9-e911-a82e-000d3a47cb1d}” />
<AppModuleComponent type=”26" id=”{b3589b99–42b9-e911-a82e-000d3a47cb1d}” />

The manual layer was the highest layer and above of the managed layer. This is a common scenario that unmanaged changes can block/freeze and stop managed changes.

Repeat the Mantra

Only changes customisations in DEV

Out of sync environments hide bugs which then appear in production. The testing you do in an environment that isn’t the same as production is not as useful for predicting how production will work.

Fix

To resolve the problem I removed the unmanaged layer which was the layer above my managed solution. When I removed the unmanaged layer, the managed solution then popped to the top I then saw the 10 views I was expecting.

In this case I knew there shouldn’t have been an unmanaged solution because no one should be making changes in QA, any changes to customisations should happen in DEV and be pushed through to QA.

Unmanaged changes in an environment are a common cause of customisations not updating and the solution layer button is good method for finding these.

Related articles on bug fixing

How development used to be (15 years ago)

Sometimes it pays to stay in bed on Monday, rather than spending the rest of the week debugging Monday’s code. — Christopher Thompson

I was listening to the Orb song little fluffy clouds and it made me think of what development was like when I was young. Whilst on lockdown and working from home, it highlighted how much the world has changed for developers.

I have been a developer and other roles for the last 20 years and the development process has changed a lot, my knowledge has grown and no code solutions are on the rise.

Starting off as a developer is tough because everything is challenging and you have no knowledge or skills to help you. The only way to improve is to try fail and learn.

You start off getting stuck every 10 minutes and keep asking your colleagues question, slowly you improve and make fewer mistakes.

Here are some of my memories from 10 to 15 years ago when I was an eager junior developer

Builds

Builds used to take hours and sometimes the best part of a day. Due to the length of the build we only released once a month or some ridiculous length of time.

I worked for smaller companies to begin with, so we didn’t have time to automate the build, this was only something I read about. Why I didn’t invest time in this area is mystery, but then no one else did either.

Build and releases were lots of manual tasks, tick lists and fixing odd bugs.

Setting up your development machine

Once you set it up and got it working, DO NOT CHANGE IT.

I remember new starters trying to setup a new machine and wasting days with it not building, no one could remember what they had done to setup their machine.

Everyone’s development machine was different.

Servers were real servers that sat somewhere in the building.

There were no online services.

Frameworks

MVC was a big thing, it was the coolest thing in town.

For Java work I started off using Struts and then moved onto Spring. I say framework, what I mean is a bunch of xml files which you configured a few things.

I remember having my Tomcat server to host the pages on my dev machine.

Merge Hell

You could hit with merge hell at any time and this was a serious problem. It would take hours to resolve all the merges.

I think the reason merging was more painful was because we merged the code every 3 or 5 days. Not every day like we do now.

Formatting

There used to be big debates on formatting your code. Do you put the bracket on a new line.

The use of Indents could involve discussions with multiple developers for hours. The end result would be everyone would go back to their own way of doing it.

Databases

We connected to databases in the code with ODBC connections. Every needed to know SQL and be willing to create tables and write queries. 

Blogs and the internet

When I started out you were lucky if there was one blog post on the topic. Now there are blogs, documentation, walkthroughs, videos and online training courses. The barrier to learning isn’t a lack of material, it’s a lack of time.

The lack of information made integrating and using new tools tricky. I remember trying to get things to work and struggling to get samples to work.

The plus side was you had to learn to do things yourself , instead of having the answers given to you and the code to copy.

Books

At the early stage of your career the books you read can make a big impact on the way you see the world.

I remember reading Effective Java and it blowing my mind how professional and skilled a software engineer could be. To design simple code which could manage dependencies and be reused was great. This book showed me how far I had to learn to become a good software engineer.

Other influential books

For more recommends read the blog post below

Recommended reading for software engineers

Visual Basic

I first saw Visual Basic at version 5 but it became big with Visual Basic 6. Visual basic mixed a GUI front end with code. You could quickly create an application quickly.

Creating web pages with Java or ASP.NET was painful, everything was very manual.

The downside is the language was like writing Excel formula’s, which would turn into chaos if you the application grew in size.

I’m sure there are some Visual Basic contractors making a good living supporting applications written in Visual Basic 6.

Writing code is difficult, writing it in Visual Basic 6 was verging on impossible.

Environment

The developers used to sit together in the same area, a developer zone. In my first few jobs I worked for companies who developed products. We didn’t have noisy open offices, we had areas where the different teams would nest.

In early teams I worked in there would be one or two moody developers who wouldn’t talk to anyone. They would be given work and a week later that work would be done and they would get more.

Every company would hire one nutter, who happened to be developer. This would be a rich source of amusement and gossip in the company.

There were no agile teams, daily stand ups and noisy open offices.

Working from home

This didn’t exist, it wasn’t an option. The first few jobs had desktops, and it wasn’t until 2008 I got a laptop. You can’t work from home with no internet and your computer is a big fat tower desktop machine.

Fridays

Friday was going down the pub at lunchtime. Having a pint and some bar snacks.

Conclusion

I’m not saying it was better, I am saying it was different. What strikes me is my lack of knowledge and how difficult it was to know what to learn and find resources to learn it.

Getting things to work was a fear and a motivator, it shows how reliant we have become with the internet. It highlights how great the developer community is and how much knowledge we share with each other.

Dynamics 365 and creating a reporting database

Data migration takes at least 4 times longer than everyone estimates #HoskWisdom

It’s a common requirement to create a different data source for reporting this allows data to be aggregated and reduces the load on your production instance of Dynamics 365.

This post will look at the options to replicate your Dynamics 365 database to a SQL database.

Azure SQL database

Azure SQL is common database that is familiar with many report writers. The common requirement is to copy your Dynamics 365 database to an Azure SQL server.

What are the options for you to do this

The Data export service

The Data export service is a good choice because it’s free and can works with Dynamics 365. Choosing what entities you copy is setup with configuration with no code needed.

The Data export service is an add on that replicates your Dynamics 365 database (or CDS) into an Azure SQL Database. After the initial copy it syncronises the changes by using changing tracking.

It copies to Azure SQL subscription, or SQL Server on an Azure VM

Here are some links to learn more

Pro’s

  • It’s free
  • integrates with Dynamics 365 and Azure SQL with configuration
  • It has an API but it can work with configuration
  • Monitoring and diagnostics on sync status
  • Full initial syncronisation
  • Metadata changes are fully syncronised

Con’s

If you do any of the below in an environment, you the export profile must be deleted and recreated

  • Restore an environment.
  • Copy (either full or minimal) an environment.
  • Reset an environment.
  • Move an environment to a different country or region.
  • You need to turn on change tracking, which can slow down the system a bit I believe

KingswaySoft

KingswaySoft is a popular tool for data migration, it works with SQL server and has a connector with Dynamics. You can use KingswaySoft to synronise data to an SQL database.

Some useful articles

Pro’s

  • KingswaySoft is powerful and you can transform the data.
  • There are lots of developers with experience and the learning Kingswaysoft is quick
  • Free developer edition
  • Cheaper than Scribe

Con’s

  • Purchase a KingswaySoft licence
  • Developer to write the packages (it’s not code but does need a professional approach)
  • You need to create an SQL server with SSIS and configure firewalls
  • You need visual studio downloaded

Scribe online

Scribe is tool to integrate and syncronise business applications and it has an out of the box connector for Dynamics 365. Scribe online is a service, so you don’t need to download anything.

I haven’t used Scribe or been on a project with Scribe but people who have used it, usually recommended it.

Pro’s

  • It’s configuration
  • online service, so now need to download or have servers
  • easy to learn and use
  • no knowledge of SQL or SSIS needed

Con’s

  • Purchase licence
  • Developer to write the packages (it’s not code but does need a professional approach)
  • no knowledge of SQL or SSIS needed
  • Can be slow sometimes and no way to speed this up
  • Costly. At least 4 times more expensive than kingswaysoft

Scribe versus KingswaySoft versus Data Export service

It seems the Data Export service is the best choice and it is if you only want to replicate your Dynamics 365 database. Kingswaysoft and Scribe can do it and a whole lot more. KingswaySoft and Scribe can integrate between systems with powerful transformation functionality.

In the scenarios where integration has been needed Kingswaysoft has been chosen over Scribe because Kingswaysoft is a lot cheaper and the project has had developers on it. The big benefit of Scribe is you don’t need to know SQL or SSIS.

I haven’t used Scribe or been on a project with Scribe but anyone who has usually recommended it.

Data migration and integration always take longer than you think and is more important than most realise.

Other options

Microsoft Power Automate and Logic apps are no code tools to integrate Microsoft business applications. These are power no code solutions with Azure capacity. The connectors and improvements Microsoft have made in Power Automate has made it a powerful tool.

What use to take C# code and a week, can now be done in a few hours with Power Automate and Logic apps. If there are a lot of integrations, this choice might lead to more maintenance and there is some missing functionality which could make it unsuitable.

Azure Data Factory

Azure data factory is a is Azure cloud ETL service that can run SSIS packages. It’s a service so you don’t need to worry about any infrastructure. It has a code free UI, it’s configure rather than code.

It’s offers Azure scale and connects with Microsoft services out of the box. The pricing seems competitive and based on compute/usage.

The downside is few developers have experience using it, so there would be upskilling involved.

Azure Data lakes

Microsoft are investing in Data lakes, making it easy to export to and promoting it. Azure Data Lakes are the direction of travel and I have the feeling Microsoft are going to invest and make this attractive for people to use.

Exporting Common Data Service data to Azure Data Lake

Azure Data lakes are made for large datasets and as we capture more data, it’s likely we will need to upgrade the tools we use.

I haven’t used a Data Lake yet but I believe this will change going forwards, so it’s something to watch and investigate if you have time.

20 things you know only if you have worked on an IT project

Plans are worthless. Planning is essential~ Dwight D. Eisenhower

  1.  The road to a late project is paved with extra requirements
  2. Trying to cleanse data in a legacy system is as easy as staring at the sun
  3. ERD’s are always wrong but no one wants to fix it
  4. 5 minutes after creating a project plan it’s out of date
  5. The sales team always over sold the solution
  6. Data migration takes at least 4 times as estimated
  7. I have seen developers 99% finished on development and the last 1% take as long as the first 99% 🙂
  8. There are many projects Agile in name but chaos in reality
  9. Adding more people to a project can make it go slower
  10. Code is created by trial, error, error, error, error, error, error, success, error, success
  11. All bugs are blamed on developers who have left the project
  12. Projects are a team sport, played by individuals
  13. 50 percent of problems on IT projects are people, the other 80 percent are estimates and expectations
  14. Assumptions cause bugs, incorrect requirements and wrong estimates
  15. Estimates are considered commitments by management, best guesses by developers
  16. Nothing integrates out of the box
  17. Every project is different. What worked on previous projects might not work on current project.
  18. The solution you end with is never the solution designed at the start
  19. Customer don’t know what they want until you show them what you done, then they know it’s not that
  20. When trust is gone, project failure won’t be long

Articles you might like

Why adding more people to a project doesn’t make it go faster

 

 

At no point in an IT project I have seen time made up without removing large parts of functionality. The common reaction to a project failing behind is to add more people to it but Brook’s law reminds us

“adding manpower to a late software project makes it later” Federick Brooks

This article will look at the reasons adding more people to a project does not speed it up and sometimes could slow it down.

My experience

I worked on a project with two scrum teams with 4 Dynamics developers in each team. The project was progressing, but the teams could have been working more efficiently. The plan was to continue to improve the output but the client doubled the scrum teams from 2 to 4 teams. This slowed down the 2 teams and the project.

Onboarding the 8 new developers took 2 months, this needed people from the existing teams to knowledge share and answer questions about processes and functionality. The onboarding meant we had 2 slow teams and the two previous teams slowed down as they answered questions from the new developers. The increase communication between 4 teams took more effort and was not always effective. The increase in work attempted, needed the teams to work closely because functionality overlapped between the team, this often caused functionality to not work or bugs to appear.

On the same projects an integration team and data migration team of 8 people joined, the new additions had a minimal effect on the existing teams.

This article will look into the reasons to explain why adding more people can slow down a project and why sometimes it doesn’t.

Project estimation

IT projects are not a place you can make up time, the more likely scenario is it takes more time than you estimated. Estimates are done using high level requirements which lack details of the required solution, when the details become clear they add more time to the project. Estimates are optimistic and assume no problems will happen on a project (technical problems, people leaving, slow decisions, no SME’s, etc), the result is estimates are underestimates.

The bidding process effects estimates on a project, usually the lowest bid wins because it seems the cheapest but not all bids are equal. Sometimes the lowest bid misses parts of the solution like DevOps, testing or people from the team. During the project the missing requirements pop up and have to be paid for.

Cost should not be the deciding factor because IT projects are long-term investments where quality, expertise and experience of delivering on time should be considered. 

Do you want to build the solution fast or do you want to build it right?

The reasons you can’t make up time because the common method to making up time, involves adding more people to a project, which brings Brook’s law . Brook’s law can be outrageously simplified to these reasons

  1. It takes some time for the people added to a project to become productive. Brooks calls this the “ramp up“ time. Software projects are complex engineering endeavors, and new workers on the project must first become educated about the work that has preceded them; this education requires diverting resources already working on the project, temporarily diminishing their productivity while the new workers are not yet contributing meaningfully. Each new worker also needs to integrate with a team composed of several engineers who must educate the new worker in their area of expertise in the code base, day by day. In addition to reducing the contribution of experienced workers (because of the need to train), new workers may even make negative contributions, for example, if they introduce bugs that move the project further from completion.
  2. Communication overhead increases as the number of people increases. Due to combinatorial explosion, the number of different communication channels increases rapidly with the number of people.[3] Everyone working on the same task needs to keep in sync, so as more people are added they spend more time trying to find out what everyone else is doing.
  3. Adding more people to a highly divisible task, such as cleaning rooms in a hotel, decreases the overall task duration (up to the point where additional workers get in each other’s way). However, other tasks including many specialties in software projects are less divisible; Brooks points out this limited divisibility with another example: while it takes one woman nine months to make one baby, “nine women can’t make a baby in one month”.

The key reason to why adding more people doesn’t make a project go faster is new people need to onboard onto a project. It takes 1 or 2 months for someone to become fully productive, they slow down the fully productive people working on the project. The initial effect of adding more people onto a project is to slow down the original team, this may be short-term pain for longer term gain, except this isn’t factored into adding more people onto a project. It results in not making the project quicker, but adding more cost.

What the adding more people plan fail to take into account is

  • Solution dependency
  • Complexity
  • Communication overhead
  • Leadership

Solution dependency

Solutions consist of multiple customisations, many dependent on other customisations. e.g. you can’t create a login page without setting up the web server and database to save the user details and passwords.

You can’t scale people on a project if the functionality is sequential or has dependencies and the work involves developers overlapping. When you create multiple customisations and/or integrations concurrently, teams will conflict and overwrite each other’s changes. 

The assumptions individuals make can be incorrect, creating bugs which are found in integration tests.

I have seen small scale examples of Dynamics professionals working on the same entity and updating forms, plugins, workflows and Javascript at the same time. The customisations overwrote and when tested produce unexpected results. What would have taken one person 1 day to create the customisations, took 2 people 3 frustrating days.

Try to imagine assigning a team of individual to write a fiction book, each writing a different chapter. The difficultly of splitting up the book into each separate chapters that depends on the chapters before it, makes it almost impossible to write all the chapters at once.

Complexity

Solution complexity increases when you have multiple customisations working together that are dependent on each other or interact. You need to design the whole solution and separate developers to create individual components and understand the whole solution.

Adding more people to a project is effective if the solutions are not interdependence. The more complex the solution, the harder it is for multiple developers to work on related customisations at the same time. The more integrated the solution, the greater the complexity and the slower multiple developers work.

Individuals creating customisations have to understand how the business works and how the whole solution works. Most individuals in a business usually only understand a small part, its difficult to understand the business, technical solution and how both fit together.

The increased complexity makes testing harder and results in more bugs where a single component is changed without considering the effects on the whole system.

back to writing the book with each person writing a chapter at the same time. The complexity of getting all plots and character arcs with no errors would be challenging.

“As a system becomes increasingly complex, the chance of failure increases. A plan can be thought of as a system. he more components you have, the more chances that something will go wrong in one of them, causing delays, setbacks, and fails in the rest of the system. Even if the chance of an individual component failing is slight, a large number of them will increase the probability of failure.” Shane Parish — Unlikely Optimism: The Conjunctive Events Bias

Communication overhead

The more people you add to a project the more communication and meetings you need to collaborate successfully. Sharing information, getting updates, agreeing on a plan and understanding all the different perspectives will be time consuming.

Any changes in solution design or best practices need to be communicated with everyone on the project.

Communication costs and miscommunication slow down the output on the project.

Leadership

IT Projects are a team sport and good teams need good leaders. It can be difficult to quantify or measure leaders because they don’t create an output. If you have worked on a project with bad leaders you can see the confusion, bad decisions and lower morale to work out the effect.

Adding more people costs more money, adds communication difficulties and increases the complexity of the project. .

Good leaders do

  • Empower and encourage other members of the team
  • They are umbrella’s — they shield the team from distraction, noise and interference
  • They over communicate and keep the team informed
  • Push back on the customer when needed
  • Manage the different characters
  • Give clear direction
  • Control the scope
  • Solve problems and improve the project
  • Listen to people
  • Know when to have an opinion
  • energise the team and motivate individuals

What good leaders don’t do

  • Add lots of people to a project hoping it will make it go faster
  • Side with the customer when they are wrong
  • Agree to impossible demands
  • Sit there and do nothing

Summary

The method commonly used to estimate projects is to split up the work into small focused parts and estimate them separately, no effort to estimate the costs of complexity of the whole solution. 

Communication, leadership, complexity and interdependent development are rarely factored into estimates. Just because it’s difficult to estimate these, its not an excuse for not trying.

The inside view leads us to overestimate our abilities and underestimate the work. Read more Why IT projects estimates are wrong

High-level requirements create high level solutions and these expand in complexity when the lower level requirements are added. Low level requirements can break initial solutions and lead to the project running behind schedule. The common response to this problem is to add more people to a project, this can slow the project down whilst costing more money.

I will leave you with this Fred Brook’s quote

How does a project get to be a year late? One day at a time. Fred Brooks

Related blogs

Advice to Dynamics 365 developers who want to improve

“If you always do what you’ve always done, you’ll always be where you’ve always been.” —T.D. Jakes

It’s a great time to be a Dynamics 365 and business apps developer with opportunities to take on enterprise projects with companies digitising legacy applications. Complex projects need developers who do more than write code and create customisations.

This article is the advice I give to developers who want to work on large projects. I wrote a similar post but less technical for functional consultants who want to become Dynamics developers.

Ask Hosk — I’m a functional consultant who wants to become a Dynamics developer

Enterprise projects are different

When you have multiple teams of developers working on a project, standards, consistency and quality become important because technical debt and chaos will erupt if not controlled. On large projects DevOps becomes a necessity, giving feedback quickly to anyone who has broken the solution.

If you have 10 people creating customisations and code, to find the problem quickly, you want to look in the smallest change set possible. Change set is the group of changes made since the build worked.

These processes reduce the number of changes you need to look at

  • Unit test — problem just in your code
  • Continuous integration — problem in last individual change
  • Nightly build — Problem in one days’ changes
  • Weekly build — Problem in weeks changes

Releasing once per sprint — 2–3 weeks would include lots of changes, making finding, diagnosing and resolving the problem difficult. This finds the the bugs in one piece of functionality, not regression testing or bugs on related functionality (unless you have automated tests).

On enterprise projects you want to reduce complexity of customisations/code with standards, quality and testing. The nearer you find the bug to when and where it was created, the quicker you resolve it and the less people involved.

Be a Software engineer

Developers concentrate on code/customisations, software engineers understand the entire development process.

There is more to creating solutions than creating code/customisation, such as

  • Building environments — DevOps
  • Releasing/deploying- DevOps, Solution packager, CI/CO
  • Testing — Unit testing, integration testing, automated testing
  • Code quality — SOLID principles, design patterns, standards, processes, static analysis, solution checker (What is the solution checker)
  • Data — Moving data between environments and data migration

Read blog posts — Learn

Knowledge compounds over time, if you keep learning it becomes an advantage.

Blog posts are a great way to learn about new functionality, solutions and best practices. Reading release notes only tells you about new features but don’t help you to learn existing functionality, problem solving and coming up with the solutions. Dynamics 365, Power Platform, Azure and business apps environment is changing at a fast pace, combined with the licencing and API limitations means the solutions and approaches we use today are different from past projects and future projects.

Keep up with the changes of services, functionality and environment by reading about new features. Don’t just read, try out new functionality and gain practical knowledge, which is more valuable than theoretical knowledge.

The way I keep up is by using Twitter and the great Tweetdeck.

I recommend creating a list and adding Dynamics, Power Apps and business apps twitter accounts, like @BenHosk and @PowerObjects

Follow hastags

  • #MSDYN365
  • #Dynamics365
  • #PowerPlatform

A few posts from to help your career

Keep learning, be curious and upgrade your skills. Keep up with Dynamics 365 and Power Platform changes because it’s evolving and improving all the time.

Read books for a deeper knowledge — Recommended reading for software engineers , a good book can change the way you view the world.

Software engineering

As Microsoft Dynamics 365 has matured, it has been used to deliver bigger and more complex projects; you need to bring the best practices of software engineering to Dynamics development.

Writing code and creating customisations are less than half a developers role, work on other skills

  • Team work
  • Communication
  • Designing solutions
  • Testing code
  • DevOps and deployments
  • Gathering requirements, clarifying user stories

Developers focus on writing code and software engineers, focus on delivering quality projects. These article highlights the difference — Are you a Dynamics Craftsman or a CRM developer?

I recommend all developers read the book — Clean Code: A Handbook of Agile Software Craftsmanship It covers core skills such as naming, commenting, methods, classes, error handling, unit tests.

Solid Principles

Understand Solid principles, this helps you write quality code which is easy to read, maintain and extend. For large projects it controls dependencies, minimiseing brittle code (which breaks with minor changes) and reduces testing and maintenance.

SOLID Principles: Explanation and examples

Solid principles are simple rules to help you create quality code. They are chapters dedicated to them in Clean Architecture: A Craftsman’s Guide to Software Structure and Design.

There are articles, YouTube videos and other resources to learn the SOLID principles but be prepared to come back to them many times. The rules are simple but they are hard to implement, it takes experience and practice to learn them.

Understand the concepts of interfaces, how they work, when to use them and why they help reduce dependencies.

Design patterns

Design patterns are examples of simple code to common problems. They give developers a common language to discuss problems and solutions. Design patterns are classics and you learn by understanding them. There is a design pattern book on my post — Recommended Books for Developers

Design patterns give you a common language to use with developers e.g.

  • Singleton — creates one class only
  • Factory — A factory class for creating objects

Articles on design patterns

I recommend learning one per week or month because they can be dry. Once you learn design patterns you can see them in the code you and others have created, it helps you understand the problems and solutions.

Unit testing

You must test your own code, don’t expect testers and other people to find your bugs. Finding problems in your own code is the quickest way to resolve them, ask yourself

How do I know this code works?

Unit tests can be run by yourself, other developers and automated to run during the build. A unit test can help developers understand the code and alert you when the code is broken.

Unit tests offer mini regression test functionality and give confidence that any changes you have made, haven’t broken the code. For large projects with different developers and lots of interconnected code, unit tests help find problems in unfamilar code.

DevOps

If something can be automated it should be automated. Of things I love about DevOps is it takes away the boredom of development. Manually doing tasks such

  • Solutions
  • Data
  • Environments
  • Running unit tests
  • building solution to check you haven’t broken it
  • Releasing all your customisations to environment

There are many tasks you need to do to enable the creation and testing of customisations. The harder you make it to test and deploy customisations the less it’s done and fewer problems you find.

DevOps reduces the feedback loop and removes obstacles to testing and environment management.

DevOps isn’t a nice to have skill, it’s a necessity. The bigger the project, the more you need DevOps and knowledge of it.

Scrum/Agile

Most projects are Agile, so make sure you understand the logic behind Agile.

Looking at the ideas behind SCRUM

There are good courses on Pluralsight, YouTube and the internet.

Agile done badly can make projects harder, so spend time getting good at it.

Azure

Plugins should be short and quick, anything more complex should be running in Azure.

Azure Functions should be your choice for long running C# code, Dynamics has a 2 minute timeout limit and moving the code to Azure reduces the load on Dynamics 365.

Azure Webhooks should be on your to learn list.

Azure gives you enterprise scale, allows you to create code and customisations outside Dynamics 365.

Dynamics 365 and Power Platform

Become an expert in Dynamics 365, pass the Dynamics 365 certifications, read blogs and write blogs.

The no code solutions of Power Automate, Logic apps are powerful tools to allow quick development and integrations. The growing number of connectors means you what used to take a day with code can be done in a few hours with Power Automate. Power Automate is easier for non-coders to understand and manage because they won’t need access to Azure.

A system can be made up of multiple small focused solutions, Power Platform allows you to create these without doing anything in Dynamics 365.

Team

Raise your profile in the team by sharing knowledge, writing blog posts and presenting things learnt on the project.

Colleagues have lots to teach you from their experience and knowledge. Learn from them, improve your relationships and harness the power of the team. A group has a large shared pool of knowledge, this is more important with the growth in areas we need to understand as a Dynamics developer. Colleagues can help you learn and use new services you haave no experience with.

Keep learning, keep improving and opportunities, promotions and money will come. Knowledge compounds and the investment in yourself will bring you the most rewards.

Your colleagues are a fast way to learn and you learn by asking questions. Senior team members are in a position you want to be in and can help you get there in the most direct route.

Start with these Hosk blog posts

The code you take away as important as the code you write

“Sometimes what you don’t do is just as important as what you do.” ― Greg McKeown

The code you take away, helps as much as the code you add. Good developers remove code and improving the quality and maintainability of the code base but no one tells you that when you start your journey as a software engineer.

Code only what is necessary

Once code is released to production, you can’t take it back. Other code becomes dependent on it with components built on top of it. Taking away non-essential code becomes more difficult once it’s in production. Changes will go through the whole life cycle from Dev to Production with testing inbetween.

The more code you write, the more bugs you potentially add. Readability is reduced with more lines of code because it adds more noise to the signal. Most people don’t write simpler code with fewer lines of code because they don’t have time or the will to do it.

Any time we write code, we can create bugs. We can create bugs as easily as useful code. All code brings the promise of new functionality and the threat of bugs.

Create as little code as possible and keep it simple. The first phase of coding a solution is to get it to work, the second phase is to subtract what is not needed and improving the naming of the code. Better named code doesn’t need comments — Code should be the one version of the truth, don’t add comments

The noble art of creating customisations and delivering solutions, the creation of code is quick, compared to time maintaining the code. The more code written, the more resources and time needed to maintain that code. Experienced developers understand the wisdom of creating as little code as possible, which reduces the overhead of the code created.

This reminded me of Robert Greene’s 4th law of 48 Laws of Power Law 04: Always Say Less than Necessary

“When you are trying to impress people with words, the more you say, the more common you appear, and the less in control. Even if you are saying something banal, it will seem original if you make it vague, open-ended, and sphinx like. Powerful people impress and intimidate by saying less. The more you say, the more likely you are to say something foolish.” — Robert Greene

Eliminate non essential

Great coders eliminate all non-essential code and create code that is as simple as possible but no simpler. Don’t accept the first draft of code, improve the naming and reduce the code to the essential. You can refactor the code when it has unit tests you can see if your change has broken the code.

The good developers keep code quality high by reducing the technical debt in the system. An effective way is to control technical debt is to not put it in. Getting code to work and is the initial goal, the second stage is taking the initial code and reducing the non-essential and achieving the same outcome.

It’s a case of looking at the code and asking yourself, is this line necessary or can I do this with fewer lines? Be mindful the goal is not the least amount of lines but removing the non essential you are improving the maintainability.

Complexity

Don’t tackle complex requirements with a complex code that is is hard to read, understand and difficult to maintain. Extra code contributes to the complexity and adds to the noise.

Complexity leads to stress and increases the difficulty of managing the code. Signs of complexity are large classes and methods. Debugging which can take days because it’s spaghetti code with little separation of dependencies.

Complex requirements must be broken down into multiple focused solutions, which are used to create a great whole solution. You don’t tackle a complex solution with complex code; you break it down into smaller components.

Design patterns

Design patterns are examples of taking a complex problem and crafting a simple solution with separate dependencies.

Design patterns are clearly named, well design and simple. There is a beauty to the code of a design pattern, no line of code is wasted and every line has a purpose.

The naming of design patterns makes it obvious

  • Factory
  • Repositry
  • Singleton

The patterns and code are not the first draft but the refactored and final draft. The code has been renamed and reduced to leave only the necessary code.

Conclusion

The focus of developers is on the creation of code but the quality of code is how well developer reduce the code to only what is necessary.

Every line of code has to be maintained.

More lines of code adds to the technical debt and complexity of the whole code set.

The picture is from here

one simple tip to improve you development and project estimates

It isn’t just developer who underestimate tasks, it’s everyone and it’s all the time #HoskWisdom

When we estimate tasks, we underestimate the time it takes because we overestimate our abilities and believe our future involves fewer problems. Most estimates assume no problems, mistakes, other activities will distract us. Using the outside view (similar examples), can help avoid this bias.

This problem is not confined to developers, a common example is people underestimating how long it takes to commute to work and who are late.

Why do developers under estimate?

Happy path estimate — no meetings, problems or any other activities will impede development

There is an unwritten rule when asking a developer to estimate how long it will take, add at least 25 percent onto the estimate or 50 percent if it’s complex. I have seen spreadsheets used with a column for confidence or unknown as a percentage. This column is then used to multiple their estimate and get towards a more accurate estimate.

You are probably thinking this sounds like padding or giving extra time but consider how many times you managed to do a task without the following happening

  • Being interrupted
  • Finding extra tasks missed in your original estimate
  • Extra requirements or scope increasing
  • A technical problem
  • Finding problems in other peoples code
  • User error, adding bugs
  • Someone asked you a question or for help on a different problem
  • A meeting over running or a new meeting arranged
  • unforeseen consequences of other functionality or the whole solution

There is pressure to estimate low from the manager, the customer and the project plan.

If life was perfect, we would get the task, know everything about it, find a quiet place, do the work with no distractions. 99% of the time that happy path doesn’t happen and other activities not on the plan slow us down.

Estimates are not commitments, they are a guide to how long we think a task will take. I have seen developers 99% finished on a piece of development and the last 1% take as long as the first 99% 🙂

Everyone overestimates

“The reason is most people think of themselves as different, and better, than those around them” Daniel Gilbert

We have an above average view of our abilities. If you ask a developer if they are an above average developer, most will reply yes.

Answer yes or no to the questions below

  • I am an above average developer
  • I am an above average driver
  • my intelligence is above average
  • I am better than most at picking films to watch

If you answer yes to 3 or 4 of those you believe you are above average but statistically you are average. We believe we are above average and because we are doing it, it will go better. We overestimate how fast we will complete the work.

Another contributing factor is known as the illusion of optimism, we believe good events are more likely and bad events like likely to us. The larger the he task the greater the chance of delays.

The inside view

Making predictions using the inside view is to estimate how long a task will take using the information we have, our understanding of the problem and solution. The brain allows us to to do this fast and make an estimate we believe is accurate.

Making predictions using the inside view risks missing complexity we hadn’t considered or ignored.

The outside view is to consider the task based on statistics or how long it took similar tasks (not the estimate).

Daniel Kahneman gives an example where he and a group of people set out to write a curriculum text book, the group estimate was 2 years. When Kahneman asked a member of the team who was a curriculum expert (who had seen many text books created), how long it took other teams to create a curriculum text book the answer was different (see the quote below)

He fell silent. When he finally spoke, it seemed to me that he was blushing, embarrassed by his own answer: “You know, I never realized this before, but in fact not all the teams at a stage comparable to ours ever did complete their task. A substantial fraction of the teams ended up failing to finish the job.”

This was worrisome; we had never considered the possibility that we might fail. My anxiety rising, I asked how large he estimated that fraction was. “About 40 percent,” he said. By now, a pall of gloom was falling over the room.

“Those who finished, how long did it take them?”

“I cannot think of any group that finished in less than seven years,” Seymour said, “nor any that took more than ten.”

I grasped at a straw: “When you compare our skills and resources to those of the other groups, how good are we? How would you rank us in comparison with these teams?”

Seymour did not hesitate long this time.

“We’re below average,” he said, “but not by much.”

The story is from here Daniel Kahneman: Beware the ‘inside view’ and is an excerpt from the book Thinking, Fast and Slow. It highlights how the inside view can underestimate the time it will take and the outside view can give you an unbiased estimate.

You can watch a video of Daniel Kahneman describing this process here

Can you guess how long it took to create the book? The group (inside view) estimate was 2 years, the outside view between the minimum 7 years — maximum 10 years.

It took 8 years, 4 times the groups original estimate and within the range predicted by the outside view.

How to overcome this?

We are optimistic about our own abilities and estimates of how long it would take us but we are more accurate if we estimate how long it would take someone else (.e.g. an average person).

When you have estimated some work, get another developer to estimate how long they think you would take to do it or ask them how long it took them to do it on a different project.

Compare the estimates of similar developments on other projects or on your project. The time took to do similar developments is devoid of bias and will give you a base rate.

We think we are different and our project is different but they aren’t, so it get an outside view from someone who can step back from the project and asses it analytically.

Other reading

Ask Hosk - I’m a functional consultant who wants to become a Dynamics developer

Here is a question from one my readers which is a common question, so I am putting the answer here so lots of people can read it.

Question

“I am working as a Dynamics 365 functional consultant But I want to become a Dynamics developer. Can you please give me an idea on what are the pre-requisites for this? I have basic knowledge of C# and MSSQL”

Hosk advice

Moving from a functional consultant to Dynamics developer is an easier path with the progress of no code and low code solutions. Power Automate, Power Apps and Logic apps offer lots of ways to create customisations that would normally have taken C# code.

Become a Dynamics 365 expert

Study for the MB-200, It has free online training at the bottom of the link.

Do not cheat and download answers because if you get caught the results of all your certifications will be voided. The value of certification is in the journey of learning, not the certification.

Don’t just study by reading, get a trial of Dynamics 365 and try the functionality out. The goal is to increase your knowledge and practical experience. Dynamics 365 doesn’t work as written. There is trial and error when using functionality, this helps you learn and understand how features work.

You don’t know when to use functionality unless you understand how it works, and its limitations. People who say they will learn it when they need it are under prepared and they waste time creating solutions that look like they work but don’t.

Study other Dynamics 365 certifications and Power Platform. Be an expert in Dynamics 365, use as much of the out of the box functionality as you can — To deliver a project on time, stick to out of the box (if you can)

Other training on Microsoft Learn

The Microsoft certifications are respected by employees and show you have Dynamics 365 knowledge. A certification might get you a new role but you your experience, knowledge and skills will keep it.

Dynamics partners need their Dynamics professionals to be certified so they retain their gold partner status.

Get a trial of Dynamics 365

Knowledge of Dynamics 365 is only useful if you can create customisations in Dynamics 365 and understand how it works.

Your job will involve using Dynamics 365, knowledge helps but practical experience is necessity. You can’t learn to drive a car by reading how a car works and a driving manual. You need experience driving.

Write code

If you want to be a developer then write code and creating customisations. The more you practice this the better you will get.

You need to create plugins, custom workflows, JavaScript, Typescript in a Dynamics 365 environment and there is nothing stopping you.

When someone says they want to be a developer but hasn’t got a Dynamics 365 trial and attempted to create all the different customisations then they are not doing all they can to help themselves become a Dynamics developer.

Create customisations and write a blog post, so any potential employers can see you have the knowledge and skills to be a Dynamics developer.

CDS

Learn about the common data service and how it works with Power Apps and other Dynamics services.

Dual-write is now available, understand what it can do and how it works

Dual-write is available to integrate Dynamics applications in near time

Power BI

Power BI is the common report writing tool for Dynamics 365 services, you can download Power BI desktop and create reports. The best way to learn is to practice writing reports.

Power Platform

Power Platform is where Microsoft is investing a lot of money, so expect this to be at the core of Dynamics and Power Platform projects in the future.

  • Create Canvas apps
  • Create model driven apps
  • Create Power Automates (Flow)

PCF controls are great and offer a powerful way to customise Dynamics forms.

Power Portals

The portal functionality is powerful,popular and features on many projects. There is a configuration and liquid templates to understand. Liquid templates are a half way house between code and configuration, they are not hard to learn but they take time to learn. A useful skill to learn.

Portal pricing has changed, useful to understand

Understanding Power Apps Portals pricing and how it differs from Dynamics Portals

Azure

Learn about Azure because more functionality is created outside Dynamics 365 and Azure works nicely with Dynamics 365/CDS

Create Logic Apps

Create Azure Functions

Read blogs posts

Read these blog posts

Other Dynamics blogs here

List Top 25 CRM Blogs

Follow hastags

  • #MSDYN365
  • #Dynamics365
  • #PowerPlatform

Read books

I tell developers to start by reading Clean Code to help master the basic of coding.

Read about SOLID principles and try to implement them

Recommended reading for software engineers

Finally

There is lots to learn to become a Dynamics developer. it takes you a long time. Don’t be disheartened by this, be encouraged because if it was easy the world would be full of good Dynamics developers.

Don’t learn everything at once, keep learning new skills and functionality all the time. Pick some of the areas and focus on them first.

Being a Dynamics developer is not a skill you can master, it’s a craft you can keep improving at.

I have another post more focused on software engineering and developer skills coming up, so stay tuned.

How to convert an unmanaged solution into a managed solution in Dynamics 365?

“You see, but you do not observe.” — Arthur Conan Doyle, A Scandal in Bohemia

I had an unusual scenario where an environment was setup with an unmanaged solution and I wanted it to be a managed solution and I needed to convert it. All environments beyond Dev should be managed, so users cannot change customisations. Changes should be made in development and the other environments use the functionality but don’t change it. Stopping changes in environments stops them becoming out of sync, which can affect your testing in those different environments because you are not testing the same customisations.

Converting the solution

I imported the solution with the same publisher in a managed state; I was hoping this would just overwrite the unmanaged solution and customisations and turn them into managed. On import it threw an error and wouldn’t let me convert the unmanaged solution into a managed solution.

Why wouldn’t it let me overwrite it to managed?

I think it’s there to stop developers accidently converted their development environment into a managed environment. This would of a disaster for a development environment because you could no longer change your customisations.

If you did this, the first thing to do when you find yourself in a hole is stop digging. Microsoft backs up your environment hourly for up to 7 days in the past. If you need too you can restore to any hour in the last week. Before you get too excited, remember you would lose customisations and data. It’s very difficult to know what data has been updated and the effort to recreate it is manual, this makes the hourly backup a last option scenario.

Converting the customisations to managed

The internet offered me no help, so I got to thinking (some articles to help you think about your problems)

An unmanaged solution is a container for customisations, if you delete the unmanaged solution, it only deletes the solution. A managed solution is different, if you delete a managed solution, you delete all the data and all the customisations. 

The solutions act differently because unmanaged solutions update the default solution with unmanaged solutions. Managed solutions create another layer of customisations on top of the default solution.

Read more about solution layers

There is a solution layer button is useful to find out how many solutions and different customisations are overlapping on a particular component. This can help diagnose puzzling behaviour and when people have been updating an environment directly.

Conclusion

I deleted the unmanaged solution and then imported the managed solution and this time it worked and turned the unmanaged customisations to managed.