3rd Party Event Tracing Calls in Apigee

Apigee has information on their website which makes event tracing of calls to a 3rd party system relatively easy. But, the information is spread out over a couple of pages. To provide this functionality effectively, you’ll want to use two different features together:

• Use a PostClientFlow to ensure the event logging is performed after the response is sent to the client.
  &nsbp;
• Use a ServiceCallout Policy, with the <Response /> element removed. This will ensure the call to the 3rd party system is done as a Fire-and-Forget call, rather than one that waits for a response before continuing processing.
  
  There is a MessageLogging Policy, which is specifically designed for this logging scenario. However, the MessageLogging policy doesn’t allow for Header information to be added into the call; and there are a number of 3rd party logging systems (like Splunk) which use the Authentication header to verify the incoming caller.

The end result of making these changes looks a little like this:

The 5 steps within the workflow taht are grouped by a red box show a group of 2 service calls which are each logging to separate 3rd party systems (we wanted to compare the two products to see which would fit our needs better). In the top left red box is the complete processing time within Apigee, 78 ms. And the small red box at the bottom right (in Postman) is the amount time from the client’s perspective, just 46 ms.

To do this, you’ll want to setup a shared flow that will make the ServiceCallout's. Remember that each ServiceCallout should remove it’s <Response> element:

Once that’s in place, you’ll just need use the shared flow as part of a <PostClientFlow> within you API’s. I wish this was an element I could use within the Post-proxy Flow Hook; that way I could add it to all APIs in one place.

A communication benefit of microservices

Recently, a blog post called Monoliths are the Future caught a coworkers attention, and he had some interesting questions that stemmed from it:

You know what's interesting; I feel like we all get this sense that we _have_ to be doing microservices

Like any existing architecture is garbage and, regardless of your business or technical constraints, you are failing if you don't immediately make a wholesale switch over to microservices

And so now I feel like I see more and more articles saying, "Hey, whoever told you that you needed to stop everything and do microservices was wrong. You should take both designs into consideration and implement something that meets the needs of your business/technical constraints."

I'm just interested in where the all-or-nothing perspective leaked in; was it our perception as self-conscious technologists? Or was it click-baity writing?

For me, I think a fair share of the attention that microservices gets stems from the email that Jeff Bezo’s wrote around 2002 that was made famous by a Google+ rant from Steve Yegge. In the post he outlined Jeff Bezo’s “Big Mandate” of:

• All teams will henceforth expose their data and functionality through service interfaces.
• Teams must communicate with each other through these interfaces.
• There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
• It doesn’t matter what technology they use.
• All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

This list wasn’t shared on the internet until 2011, and SOA and microservices had already become popular at that point. But, what Bezo’s list did was create the building blocks for AWS, and AWS’s incredible success is what I consider to be one of the largest factors in people looking at and believing that microservices are the way to be successful. Many people don’t understand the reason behind AWS’s microservice based success, they simply believe the false equivalency that they will achieve AWS’ successes by using microservices.

But, for me, that list from Bezo’s captures an underlying management strategy that is far more impactful that the microservices themselves. What that list succinctly describes is that all interactions between the systems at Amazon/AWS would now be based around contracts and interfaces that are developed and managed by service providers. This means that the service provider is responsible for working with their customers to develop a usable and meaningful service contract that guarantees that if the client can provide inputs X then their system will provide service Y. They guarantee that the service will be available 24 hours a day. They guarantee that it will be usable without requiring any communication between the two separate teams, without the overhead of one-off reviews and without any specialized approval processes.

This contract-first always-available platform approach effectively breaks one of the most difficult constraints on all projects: Communication Overhead.

In The Mythical Man Month, Fred Brooks describes that when you add another person onto a project you’re also adding an exponential number of points of communication to that project team. If you had 10 people on a project and you add an 11th, then you just added 10 more avenues of communication and slow down.

Within teams that are highly effective, a significant fraction of their productivity comes from their shared knowledge about what they’re building and the goals they’re trying to achieve together. Agile seemingly builds on the back of the lessons learned from the Mythical Man Month by trying to reduce that communication overhead. Agile uses the daily stand-up to build shared knowledge and shared vision on a daily basis. The daily stand-up is where a team member can quickly ask “I want to do X because I think it will give us benefit Y. Is everyone on board with that?” Because everyone at the daily team meeting has been sharing their knowledge, that statement wouldn’t require a great deal of time explaining the context and history of how the thought came to be or why it would be beneficial. The reduction in explanation time between those team members is one of the aspects that makes that team effective.

Oppositely, when you have to communicate outside of your team, that’s when you introduce a communication constraint. Bringing another team into the conversation and bring them up to speed takes time. You have to explain to them where your team is at and get them into your teams mindset. The hardest part of that conversation is that while you’re explaining the background and reasoning for how your project got to it’s current point, the new team is going to view every decision that was made along the way through their current understanding and their guiding principles; not through your teams current understanding and guiding principles. This leads to multiple levels of slow down and overhead, not to mention the hardest thing of all: disagreement.

What Jeff Bezo’s email did was lower the overall cross team communication overhead company wide at Amazon. If your project could satisfy the requirements of interface/contract X, then the service would provide the results Y at any time of day with no waiting for two teams to find time to meet, evaluations of requirements and agreement on purpose of product, or the time consuming process of bringing individuals with differing view points into alignment.

But, many people don’t think about that side of it. I feel like the original designers of microservices have tried to stress the importance of Team Autonomy in Microservices as a key component to ensuring that each team/each service provider can create new functionality without requiring them to derive agreement and get sign-off from another team. In a monolithic database project that I have worked on, I have seen this problem occur around the very few but very important tables that multiple teams share. They require a great deal of communication overhead to ensure all teams are aware, have analyzed and reviewed, and have implemented plans to handle any consequences of an update; all done before even the first action can be taken on the update.

But, as Fred Brooks said, there is No Silver Bullet, and reducing cross team communication overhead is just one piece of a much larger and more complicated puzzle of making an effective working environment.

So, I think “the all-or-nothing perspective leaked in” because of a combination of things: AWS’s great success, Martin Fowler (et.al.) evangelizing mircoservices, and the audience that was absorbing this information not having the full perspective of what was truly driving the benefits.

As if I haven’t already given my two cents … here were my actual thoughts on the original blog post, Monoliths are the Future:

• There were a number of statements within the post (and I have not listened to the audio) which made me believe that an underlying problem the speakers were grappling with was low code quality standards and coding practices at their companies. For example, statements like “we lost all of our discipline in the monolith”, “they’re initiating things and throwing it over a network and hoping that it comes back”, and “Now you went from writing bad code to building bad infrastructure”.
  
  I believe that ensuring quality within the products and services you provide is a critical necessity for anything to be successful. In Lean Six Sigma practices, defects are one of the critical wastes. You must ensure high quality and resilient code in order to reduce the amount of time spent on rework.
  
  You don’t have to use microservices, or Gang of Four, or XYZ to ensure high quality standards; but the company you work for has to define high quality services and products as one of their highest priorities. From there, the people at the company will develop the standards, processes, and tooling to ensure that they are creating high quality products and also monitoring that those standards are upheld every day.
   
  
• The line “There are reasons that you do a microservice. So, to me a microservice makes sense in the context of…” was a silver lining to me.
  
  The writer was outlining that when he can see value with using a particular approach, then he is on board with making that approach successful. This is probably the most important aspect of choosing any architectural approach. If the people that are implementing the approach can see the value within it, then they are not only on board with making it happen, they will find ways to make it better than it was originally designed.
   
  
• There is an aspect of microservices where monolithic datastores are separated into small autonomous datastores. This separation is to improve team autonomy and lower communication overhead. But, there is a flip-side to creating those small autonomous datastores. Besides the obvious duplication of data, it also creates a new need of bringing the data back together in order to analyze it from a system wide perspective. Whenever data stores are broken apart, there is a need to create a new centralized datastore for reporting and business insights. These have most recently been coming up as Big Data, Data Lakes, and other data collectors for Business Intelligence and Data Analytics platforms.
  
  So, eventually, you always get back to a monolithic data store; but maybe not a monolithic application.
   
  
• My last thought is pretty negative. Again, I haven’t listened to the audio, so this might be completely off base. I just don’t get a strong sense that the author of the article or the speakers being quoted are really thinking about things from an overall workload productivity perspective. How are all the people of the company working together to make the company’s product? What are the processes that the company has in order to make those products? And, what are the most critical foundational principles that the company needs to do in order to make those processes as effective as possible? If you have the answers to those questions, then the question of microservices vs monoliths will have a clear answer.
  

ExceptionHandler Needed

As a follow-up to the Create a Custom ProblemDetailsFactory post, it has been discovered that a custom Exception Handler must be defined in order to use the ProblemDetailsFactory. The Exception Handler can be incredibly simple, but it must produce an IActionResult that will trigger the calling of a ProblemDetailsFactory. This can be accomplished with something as simple as this:

The controller only needs to create a Problem IActionResult to trigger the usage your ProblemDetailsFactory.
To ensure that YourBussProblemDetailsFactory is used, you can create two extension functions to use during Startup.cs’s ConfigureServices and Configure methods. That might look like this:

And the extensions functions that provide the wiring would look like the code sample below. Some noticeable pieces in the example are:

• Within the IServiceCollection extension method, .AddMvc().AddApplicationPart(thisAssembly) is used to ensure that the controller from above is included within the top level application. This is how you can add controllers from sub-libraries.
• Within the IApplicationBuilder extension method, the final wiring to setup the global exception handler is created to “/your-buss-exception-handler”.

There's an example Exception Detail Converter in a follow-up post (ExceptionDetailConverter Example).

SSH Key Auth to GitHub on Win 10 w/ VSCode

My work computer has been using SSH keys to authenticate to GitHub for a while. But I’ve slept a few nights since I set that up and I have no clear memory of what I did.

I wanted to setup my home computer the the same way and struggled to figure out how to do it. So, I thought it might be worth documenting.

The secret (I think) … install git 2.20.0 or higher

In the end, the final change that made SSH key authentication work was updating my git installation from version 2.15.0 to 2.25.0. My work computer has 2.20.0 on it, so I figure that should be the minimum level.

Here’s an outline of the things I tried and notes about them:

	Work Computer	Home Computer	Notes
git (version)	2.20.0	2.15.0 –> 2.25.0	Didn’t work with 2.15.0. Finally worked with 2.25.0.
SSH keys	I don’t remember how I generated them. I think I generated the keys using ubuntu WSL.	Copied them from my work computer to my home computer using normal NTFS system (didn’t need Git Bash, WSL, or any of those). I did register the keys using ssh-add in `Git Bash`, ‘wsl’, and using the Windows 10 ssh-add (see Notes). But, in the end, I turned off Win10’s ssh-agent service and the SSH keys continued to be used for authentication.	On my current version of Win 10, you can start an ssh-agent service in windows. Which means you don’t need to to use a bash command prompt to execute `ssh-keygen` or `ssh-add` commands. Reminder: use `ssh-add –l` to list already registered keys.
Github PAT	I never created one for this machine.	I created one for this machine, and it would work for an individual commit (username: normal github account name, password: PAT)	I don’t think this is needed.

Reducing Noise in Error Logs / Removing PS Errors

I have a nightly scheduled job which will send out a notification email if the job has an error occur anywhere within it (even when the error is handled). This job infrequently sends out the error email. However, my long history of reviewing these emails has brought me to the point where I assume the error is always:

• There was a file lock on file X when the file was being save; the function detected the error, waited a brief time period for the lock to clear and then retried the save operation successfully.

I can't remember a time when that wasn't the case. Because of this, I am finding myself less interested in actually reading the error message and desiring to simply ignore the email. But, I know that is going to lead to a situation where something unexpected will happen and I'll ignore the warning emails. Which would be a failure of the entire warning system.

So, what I have is a very narrowly defined and well known case of when the exception occurs, and I have a desire to ignore it. If I setup the code to simply suppress this error after the save operation successfully completes, then I should be able to safely reduce the amount of noise in the error messages that are sent to me. (It should still report the error if the retries never complete successfully)

This is a very common scenario: Teams setup a warning mechanism that is highly effective when a system is first built. At that time, there are a myriad of possible unforeseen errors that could occur. There also hasn’t been enough operational history to feel that the system is stable, so being notified on every potential problem is still a welcome learning experience. As those problems are reduced or eliminated it builds trust in the new system. However, it’s also very common that once a team completes a project and does a moderate amount of post deployment bug fixes, they are asked to move on and prioritize a new project. Which gives no devoted / allocated time to maintaining small and inconsistent issues that arise in the previous project(s).

Unfortunately, the side effect of not giving the time needed to maintain and pay down the technical debt on the older projects is that you can become used to “little” problems that can occur on them; including ignoring the warning messages that they send out. And this creates an effect where you can start to distrust that the warning messages coming from a system are important, because you believe that you know the warning is “little” or “no big deal”.

The best way to instill confidence in the warning and error messages produced by a system is to ensure that the systems only send out important messages, separating the Signal from the Noise.

For my scenario above, the way I’m going to do this is to prevent these handled errors from sending out notification emails. This goes against best practices because I will need to alter the global error monitor in Powershell, $global:Error. But, given that my end goal is to ensure that I only receive important error messages, this seems like an appropriate time to go against best practices.

Below is a snippet of code which can be used to remove error records from $global:Error that fit a given criteria. It will only remove the most recent entries of that error, in order to try and keep the historical error log intact.

You need to be careful with this. If the error you’re looking for occurs within a loop with a retry policy on it, then you need to keep the errors which continued to fail beyond the retry policy, and only remove future errors when the retry policy succeeded. You can better handle the retry policy situation by using the –Last 1 parameter.

Book Review?: The Unicorn Project

The Unicorn Project (amazon, audible, supplements: itrevolution) is a new book/follow up of The Phoenix Project by Gene Kim.

And, it’s much more inline with what I was expecting the The Phoenix Project to be. The Phoenix Project focused on The 3 Ways with a strong emphasis on it’s connection to Lean Management. This was done intentionally as the book was supposed to be a retelling of The Goal done with DevOps in mind. In order for The Phoenix Project to tell it’s story it needed to be told from the perspective of someone who was required to see the whole picture of the company, to facilitate understanding of The First Way. To do that, the protagonist is a high level CIO type which has overview of all IT operations in the company. This means that a lot of the day-to-day aspects of a mid-level manager or front-line implementer are glossed over. I would even describe the book as mostly focusing on The First Way (taking more than half the book to explain) and The Second and Third Way also get a bit glossed over. But, in the context of that book, it’s fine. Because “the goal” of that book is to introduce The 3 Ways and give practical examples to help them stick with the reader.

This book continues to build upon the information given in The Phoenix Project, but it presents the information in two modified ways:

• The book is from the point of view of someone who is really a mid-level manager, but the book needs to force her into a front-line implementer position from time to time. This is done to allow for more tangible day-to-day examples to be presented of what can be done.
• The details of the external world are updated to more closely match the current state of DevOps and IT work in 2018/2019. The book references some of the newer capabilities in NoSQL databases, functional programming, and automated testing.

If The Phoenix Project was about describing The 3 Ways. Then this book is about describing The Five Ideals (which are still Lean aligned + some other ideas) :

• The First Ideal – Locality and Simplicity (Remove Mudas/Wastes)
• The Second Ideal – Focus, Flow, and Joy (A lot of things have to work together to get to Joy, Project to Product)
• The Third Ideal – Improvement of Daily Work (Kaizen)
• The Fourth Ideal – Psychologic Safety (Generative Culture, Complex Systems Fail)
• The Fifth Ideal – Customer Focus (Lean Startup, Lean Six Sigma, OKR)

They are all very useful ideals, but the book seemed to fall prey to glossing over details on how to achieve them. As mentioned earlier, there was a similar problem in The Phoenix Project. An example in this book is that our protagonist, Maxine, worked with her team to help define that a Continuous Integration (build) system needs to run Unit Tests in order to verify that each check-in of code doesn’t break the overall functionality. This is introduced as a new concept for their team. The night she introduces the idea, she falls ill and is sick for the next three days. When she returns to work, everyone on the team is writing well designed unit tests and the system has full code coverage. What?! To get a team that has never used unit tests to (a) embrace the value that unit tests provide, (b) take the time to learn a unit testing pattern that isn’t brittle, (c) create meaningful code coverage takes weeks and (d) involves a great deal of mentoring, code review, and will cause frustrations about where your teams time is most valuably spent. But, for this book, it can happen overnight with no negative consequences or trade-offs.

One thing that I really like about this book is that it is trying to take years and years of knowledge and distil it into an easily understandable and entertaining format that might get someone interested in learning more. Hopefully, it encourages anyone that enjoys the book to continue reading. The books publisher, itrevolution.com,  has a number of other books that dive deeper into the subject matter of DevOps and Business Management. By reading or listening to any of their books, you will find a long list of referenced material to continue learning from.

Book Review?: Team Topologies

Matthew Skelton created a blog post/website describing team topologies a few years ago, and with the help of others has created a book from it (Amazon, Audible).

The book breaks down the teams into 4 architypes (supplemental materials at itrevolution.com):

• Stream Aligned Team
• Enabling Team
• Complex Subsystem Team
• Platform Team

And 3 Interaction Modes:

• Collaboration
• X-as-a-Service
• Facilitating

The book will help describe the team types and what their usage patterns are.

I ran across the website which shows the Anti-Patterns and Patterns of Team Topologies about a year ago. I was trying to answer questions from coworkers about “What should our teams look like if we do DevOps”, and the website left me with a lot of guess work into the details of each of the patterns. I don’t think the original intent of the website was to be confusing, but it explored an area that I wanted to know more about so it left me with more questions than answers. And, this book didn’t seem to directly answer those questions.

It seemed to expand on the original work with new ideas and new thoughtful constructs. The team architypes lined up well with what I had seen in my own work and also lined up well with my prior understanding of DevOps / Lean Project Management methodologies. All of which are well described in the books from itrevolution.com and other publishers. (This is not an advertisement, these books have well established and useful knowledge within them.)

For the most part, the Stream Aligned Teams make a lot of sense to me because I have seen working example of them. At work, we have multiple teams which sit within distinct departments and work on the applications/projects/products for those departments.

However, the Enabling and Platform teams seemed like they could be much more intertwined than what the book described. I personally feel like I work within a platform team, but the platforms that I help provide are only useful if (1) they are developed in collaboration with a Stream Aligned Team and (2) the platform product in demonstrated/shared/knowledge-transferred to all Stream Aligned Teams. To me, #2 seems like the function of an Enabling Team. So, it feels like a company that wants to establish a permanent Enabling team would need to contain members that were in constant rotation with the Platform teams in order to:

• Keep Enabling Team members up to date on the latest utilities developed by the Platform teams
• Keep the Enabling Team members skills sharp on implementation details of the Platform team products
• Allow for Enabling Team members to develop new functionality into the Platform without having to cross a communication/trust boundary (fighting Conway’s Law)

I think that the Enabling and Platform teams might have been separated in the book due to the need for the Enabling team members to have a higher degree of interpersonal communication and collaborative skills. To be very blunt, the Silicon Valley architypes of system administrators didn’t come out of thin air (1, 2) and Platform teams would need good System Administrators.

So, maybe it’s the more evangelist personality types of the Platform Teams that can rotate into the Enabling Teams.

The only team type that I never fully connected with was the Complex Subsystem Team. I thought of some potential example projects which could fit the description. But the teams were adhoc and my example cases never felt quite right. Maybe I just don’t work someplace that has difficult enough problems to require such teams. The book is very clear that these teams are optional and would not be needed for all work environments.

In the end, I think the book adds to the overall body of literature on DevOps, and it can really help a group if they are scratching their heads about what sort of team structure could help improve their organizational structure.