A communication benefit of microservices

Recently, a blog post called Monoliths are the Future caught a coworkers attention, and he had some interesting questions that stemmed from it:

You know what's interesting; I feel like we all get this sense that we _have_ to be doing microservices

Like any existing architecture is garbage and, regardless of your business or technical constraints, you are failing if you don't immediately make a wholesale switch over to microservices

And so now I feel like I see more and more articles saying, "Hey, whoever told you that you needed to stop everything and do microservices was wrong. You should take both designs into consideration and implement something that meets the needs of your business/technical constraints."

I'm just interested in where the all-or-nothing perspective leaked in; was it our perception as self-conscious technologists? Or was it click-baity writing?

For me, I think a fair share of the attention that microservices gets stems from the email that Jeff Bezo’s wrote around 2002 that was made famous by a Google+ rant from Steve Yegge. In the post he outlined Jeff Bezo’s “Big Mandate” of:

• All teams will henceforth expose their data and functionality through service interfaces.
• Teams must communicate with each other through these interfaces.
• There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
• It doesn’t matter what technology they use.
• All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

This list wasn’t shared on the internet until 2011, and SOA and microservices had already become popular at that point. But, what Bezo’s list did was create the building blocks for AWS, and AWS’s incredible success is what I consider to be one of the largest factors in people looking at and believing that microservices are the way to be successful. Many people don’t understand the reason behind AWS’s microservice based success, they simply believe the false equivalency that they will achieve AWS’ successes by using microservices.

But, for me, that list from Bezo’s captures an underlying management strategy that is far more impactful that the microservices themselves. What that list succinctly describes is that all interactions between the systems at Amazon/AWS would now be based around contracts and interfaces that are developed and managed by service providers. This means that the service provider is responsible for working with their customers to develop a usable and meaningful service contract that guarantees that if the client can provide inputs X then their system will provide service Y. They guarantee that the service will be available 24 hours a day. They guarantee that it will be usable without requiring any communication between the two separate teams, without the overhead of one-off reviews and without any specialized approval processes.

This contract-first always-available platform approach effectively breaks one of the most difficult constraints on all projects: Communication Overhead.

In The Mythical Man Month, Fred Brooks describes that when you add another person onto a project you’re also adding an exponential number of points of communication to that project team. If you had 10 people on a project and you add an 11th, then you just added 10 more avenues of communication and slow down.

Within teams that are highly effective, a significant fraction of their productivity comes from their shared knowledge about what they’re building and the goals they’re trying to achieve together. Agile seemingly builds on the back of the lessons learned from the Mythical Man Month by trying to reduce that communication overhead. Agile uses the daily stand-up to build shared knowledge and shared vision on a daily basis. The daily stand-up is where a team member can quickly ask “I want to do X because I think it will give us benefit Y. Is everyone on board with that?” Because everyone at the daily team meeting has been sharing their knowledge, that statement wouldn’t require a great deal of time explaining the context and history of how the thought came to be or why it would be beneficial. The reduction in explanation time between those team members is one of the aspects that makes that team effective.

Oppositely, when you have to communicate outside of your team, that’s when you introduce a communication constraint. Bringing another team into the conversation and bring them up to speed takes time. You have to explain to them where your team is at and get them into your teams mindset. The hardest part of that conversation is that while you’re explaining the background and reasoning for how your project got to it’s current point, the new team is going to view every decision that was made along the way through their current understanding and their guiding principles; not through your teams current understanding and guiding principles. This leads to multiple levels of slow down and overhead, not to mention the hardest thing of all: disagreement.

What Jeff Bezo’s email did was lower the overall cross team communication overhead company wide at Amazon. If your project could satisfy the requirements of interface/contract X, then the service would provide the results Y at any time of day with no waiting for two teams to find time to meet, evaluations of requirements and agreement on purpose of product, or the time consuming process of bringing individuals with differing view points into alignment.

But, many people don’t think about that side of it. I feel like the original designers of microservices have tried to stress the importance of Team Autonomy in Microservices as a key component to ensuring that each team/each service provider can create new functionality without requiring them to derive agreement and get sign-off from another team. In a monolithic database project that I have worked on, I have seen this problem occur around the very few but very important tables that multiple teams share. They require a great deal of communication overhead to ensure all teams are aware, have analyzed and reviewed, and have implemented plans to handle any consequences of an update; all done before even the first action can be taken on the update.

But, as Fred Brooks said, there is No Silver Bullet, and reducing cross team communication overhead is just one piece of a much larger and more complicated puzzle of making an effective working environment.

So, I think “the all-or-nothing perspective leaked in” because of a combination of things: AWS’s great success, Martin Fowler (et.al.) evangelizing mircoservices, and the audience that was absorbing this information not having the full perspective of what was truly driving the benefits.

As if I haven’t already given my two cents … here were my actual thoughts on the original blog post, Monoliths are the Future:

• There were a number of statements within the post (and I have not listened to the audio) which made me believe that an underlying problem the speakers were grappling with was low code quality standards and coding practices at their companies. For example, statements like “we lost all of our discipline in the monolith”, “they’re initiating things and throwing it over a network and hoping that it comes back”, and “Now you went from writing bad code to building bad infrastructure”.
  
  I believe that ensuring quality within the products and services you provide is a critical necessity for anything to be successful. In Lean Six Sigma practices, defects are one of the critical wastes. You must ensure high quality and resilient code in order to reduce the amount of time spent on rework.
  
  You don’t have to use microservices, or Gang of Four, or XYZ to ensure high quality standards; but the company you work for has to define high quality services and products as one of their highest priorities. From there, the people at the company will develop the standards, processes, and tooling to ensure that they are creating high quality products and also monitoring that those standards are upheld every day.
   
  
• The line “There are reasons that you do a microservice. So, to me a microservice makes sense in the context of…” was a silver lining to me.
  
  The writer was outlining that when he can see value with using a particular approach, then he is on board with making that approach successful. This is probably the most important aspect of choosing any architectural approach. If the people that are implementing the approach can see the value within it, then they are not only on board with making it happen, they will find ways to make it better than it was originally designed.
   
  
• There is an aspect of microservices where monolithic datastores are separated into small autonomous datastores. This separation is to improve team autonomy and lower communication overhead. But, there is a flip-side to creating those small autonomous datastores. Besides the obvious duplication of data, it also creates a new need of bringing the data back together in order to analyze it from a system wide perspective. Whenever data stores are broken apart, there is a need to create a new centralized datastore for reporting and business insights. These have most recently been coming up as Big Data, Data Lakes, and other data collectors for Business Intelligence and Data Analytics platforms.
  
  So, eventually, you always get back to a monolithic data store; but maybe not a monolithic application.
   
  
• My last thought is pretty negative. Again, I haven’t listened to the audio, so this might be completely off base. I just don’t get a strong sense that the author of the article or the speakers being quoted are really thinking about things from an overall workload productivity perspective. How are all the people of the company working together to make the company’s product? What are the processes that the company has in order to make those products? And, what are the most critical foundational principles that the company needs to do in order to make those processes as effective as possible? If you have the answers to those questions, then the question of microservices vs monoliths will have a clear answer.
  

AWS API Key Exposed in github

AWS doesn’t want you to accidentally expose your API Key through github. And as a courtesy, they scan the creation of all new repositories in github for AWS Keys. Supposedly it’s done with this tool: truggleHog. And I couldn’t be happier that they do this! Not only for their own sake, but also the piece of mind of their customers.

Some months back I was asked to update some secrets because of exactly that scenario. A programmer had accidently checked in their AWS Keys into a public github repository and AWS had emailed our account managers to report that:

1. The key had been detected in repository X …
2. And, it had been deactivated on AWS

That was fantastic. And they detected and shutdown the key in under an hour of the key being exposed on github.

Now comes the truly amazing part …

The reason AWS does this is because malicious parties are also scanning creation of all new repositories in github and they are also looking for the same keys. And, a malicious group had found our keys before AWS deactivated them.

The automated attacker used the keys to spin up instances of x-large EC2 instances with plenty of GPU/CPU power and SSD drives. We assume the instances they created where then fitted with crypto-mining software and they went to town.

After receiving AWS’ email about the keys being disabled, our AWS contact point looked through our account and found the illegitimate EC2 instances and started killing them. So, cased closed right? Nope …

Our AWS manager shut down the EC2 instances that were in our primary region, in US-WEST-2. But, we all forgot to check the other regions. And the attackers had spun up identical stacks in all regions of AWS. The next morning we awoke to the realization that it was probably running in other regions and built a script to shut them down as quickly as possible.

All told, the 12 hour period that the EC2 instances were running ended up being over $5000 in charges.

But, AWS to the rescue again. Because it was the first time this had happened, AWS forgave the bill with two conditions:

• If it happens again, we pay for it.
• We needed to setup Billing Alarms in case any of our services (legitimate or not) starts to create charges that we are not comfortable with.

All-in-all, AWS is really trying to help out their customers; and I kind of want to give them a big hug.

AWS ALB Price Planning w/ IIS : Add IisLogWebAppId

This post continues the series from AWS ALB Price Planning w/ IIS : Grouped Sites.

This doesn’t really help figure out much in the larger picture. But, I wanted to separate out statistics about the Web API applications from the normal web applications. Web API applications are strong candidates for rewrites as Serverless ASP.NET Core 2.0 Applications on Lambda Functions. Changing these applications to Lambda Functions won’t reduce the cost of the ALB as they will still use host names that will be serviced by the ALB. But, this will help figure out the tiny tiny costs that the Lambda Functions will charge each month.

This is just an intermediary step to add WebAppId’s to all of the requests.

Background Info

Instead of adding a WebAppId column onto the IisLog, I’m going to create a new table which will link the IisLog table entries to the ProxyWebApp table entries. The reason for this is that the IisLog table has 181,507,680 records and takes up 400 GB of space on disk. Adding a new column, even a single integer column, could be a very dangerous operation because I don’t know how much data the system might want to rearrange on disk.

Plan of Action and Execution

Instead, I’m going to

1. Add a WebAppId int Identity column onto table dbo.ProxyWebApp. The identity column won’t be part of the Primary Key, but it’s also a super tiny table.
2. Create a new table called dbo.IisLogWebAppId which takes the Primary Key of table dbo.IisLogWebAppId and combines it with WebAppId.
3. Create a script to populate dbo.IisLogWebAppId.
4. Create a stored procedure to add new entries nightly.

The scripts are below, but I think it’s worthwhile to note that the script to populate dbo.IisLogWebAppId took 4h57m to create on 181,507,680 records which was 15 GBs of disk space.

AWS ALB Price Planning w/ IIS : Grouped Sites

This post continues the series from AWS ALB Price Planning w/ IIS : Rule Evaluations Pt. 2.

Having all the data in IIS and generating out all the hourly LCU totals helps define what the monthly charges could be. But, my expectations are that I will need to split the 71 DNS host names over multiple ALBs in order to reduce the total cost of the LCUs. My biggest fear is the Rule Evaluation Dimension. The more the host names on a single ALB, the more likely a request will go past the free 10 rule evaluations.

To do this, I need to build a script/program that will generate out possible DNS host name groupings and then evaluate the LCUs based upon those groupings.

In the last post I had already written a simple script to group sites based upon the number sub-applications (ie. rules) they contain. That script didn’t take the next step, which is to evaluate the combined LCU aggregates and recalculates the fourth dimension (the Rule Evaluation LCU).

But, before that …

The Full Database Model

So, instead of having to comb through the previous posts and cobble together the database. I think the database schema is generic enough that it’s fine to share all of it. So …

• Full database object script

New Additions:

• ALBGrouping
  
  This is the grouping table that the script/program will populate.
  
• vw_ALB*
  
  These use the ALBGrouping table recalculate the LCUs.
  
• usp_Aggregate_ALBLCUComparison_For_DateRange
  
  This combines all the aggregates (similar to vw_ProxySiteLCUComparison). But, the way the aggregation works, you can’t filter the result by Date. So, I needed a way to pass a start and end date to filter the results.

ALB Grouping Results

Wow! It’s actually cheaper to go with a single ALB rather than even two ALBs. It way cheaper than creating individual ALBs for sites with more than 10 sub-applications.

I wish I had more confidence in these numbers. But, there’s a really good chance I’m not calculating the original LCU statistics correctly. But, I think they should be in the ball park.

I have it display statistics on it’s internal trials before settling on a final value. And, from the internal trials, it looks like the least expensive options is actually using a single ALB ($45.71)!

Next Up, AWS ALB Price Planning w/ IIS : Add IisLogWebAppId.

AWS ALB Price Planning w/ IIS : Bandwidth

This post continues the series about from AWS ALB Price Planning w/ IIS : Active Connections. Here are couple things to note about the query to grab Bandwidth:

• This is one thing that IIS logs can accurately evaluate. You can get the number of bytes sent and received through an IIS/ARR proxy server by turning on the cs-bytes and sc-bytes W3C logging values. (see screen shot below)
• AWS does that pricing based on average usage per hour. So, the sql will aggregate the data into hour increments in order to return results.

Bandwidth SQL Script

Graphing the output from the script shows:

• Mbps per Hour (for a month)
• The jump in the average number of new connections in the beginning of the month corresponded to a return of students to campus. During the beginning of the month, school was not in session and then students returned.
• The dip at the end of the month has to do with a mistake I made loading some data. There is one of data that I forgot to import the IIS logs, but I don’t really want to go back and correct the data. It will disappear from the database in about a month.
• Mbps per Hour LCUs
• This is a critical number. We put 215+ websites through the proxy server. The two AWS ALB dimensions that will have the biggest impact on the price (the number of LCUs) will be the Bandwidth usage and the Rule Evaluations.
• I’m very surprised that the average LCUs per hour for a month is around 2.3 LCUs, which is very low.

Next Up, AWS ALB Price Planning w/ IIS : Rule Evaluations.

AWS ALB Price Planning w/ IIS : Active Connections

This post continues the series about from AWS ALB Price Planning w/ IIS : New Connections. Here are a couple things to note about the query to grab Active Connections:

• This query is largely based on the same query used in AWS ALB Price Planning w/ IIS : New Connections [link needed].
• It’s slightly modified by getting the number of connections per minute rather than per second. But, all the same problems that were outlined in the last post are still true.
• AWS does their pricing based on average usage per hour. So, the sql will aggregate the data into hour increments in order to return results.

Active Connections SQL Script

Graphing the output from the script shows:

• # of Active Connections per Minute by Hour (for a month)
• The jump in the average number of new connections in the beginning of the month corresponds to a return of students to campus. During the beginning of the month, school was not in session and then the students returned.
• The dip at the end of the month has to do with a mistake I made loading some data. There is one day of IIS logs that I forgot to import, but I don’t really want to go back and correct the data. It will disappear from the database in about a month.
• # of Active Connections per Second by Hour Frequency
• This doesn’t help visualize it as well as I would have hoped. But, it does demonstrate that usually the number of active connections per minutes will be less than 3000; so it will be less than 1 LCU (1 LCU = 3000 active connections per minute).

Next Up, AWS ALB Price Planning w/ IIS : Bandwidth.

AWS ALB Price Planning w/ IIS : New Connections

AWS ALB Pricing is not straight forward but that’s because they are trying to save their customers money while appropriately covering their costs. The way they have broken up the calculation for pricing indicates that they understand there are multiple different reasons to use an ALB, and they’re only gonna charge you for the feature (ie. dimension) that’s most important for you. That feature comes with a resource cost and they to charge you appropriately for the resource that’s associated with that feature.

Today, I’m going to (somewhat) figure out how to calculate one of those dimensions using IIS logs from an on-premises IIS/ARR proxy server. This will help me figure out what the projected costs will be to replace the on-premise proxy server with an AWS ALB. I will need to calculate out all the different dimensions, but today I’m just focusing on New Connections.

I’m gonna use the database that was created in Putting IIS Logs into a Database will Eat Disk Space. The IisLog table has 9 indexes on it, so we can get some pretty quick results even when the where clauses are ill conceived. Here are a couple things to note about the query to grab New Connections:

• As AWS notes, most connections have multiple requests flowing through them before they’re closed. And, IIS logs the requests, not the connections. So, you have to fudge the numbers a bit to get the number of new connections. I’m going to assume that each unique IP address per second is a “new connection”.
• There are all sorts of things wrong with this assumption:
• Browsers often use multiple connections to pull down webpage resources in parallel. Chrome uses up to 6 at once.
• I have no idea how long browsers actually hold open connections.
• Some of the sites use the websocket protocol (wss://) and others use long polling, so there are definitely connections being held open for a long time which aren’t being accounted for.
• And I’m probably going to reuse this poorly defined “fudging” for the number of Active Connections per Minute (future post / [link needed]).
• When our internal web app infrastructure reaches out for data using our internal web services, those connections are generally one request per connection. So, for all of the requests that are going to “services”, it will be assumed each request is a new connection.
• AWS does their pricing based on average usage per hour. So, the sql will aggregate the data into hour increments in order to return results.
• Sidenote: Because the AWS pricing is calculated per hour, I can’t roll these numbers up into a single “monthly” value. I will need to calculate out all the dimensions for each hour before having a price calculation for that hour. And, one hour is the largest unit of time that I can “average”. After that, I have to sum the results to find out a “monthly” cost.

New Connections SQL Script

Graphing the output from the script shows:

• # of New Connections per Second by Hour (for a month)
• The jump in the average number of new connections in the beginning of the month corresponds to a return of students to campus. During the beginning of the month, school was not in session and then the students returned.
• The dip at the end of the month has to do with a mistake I made loading some data. There is one day of IIS logs that I forgot to import, but I don’t really want to go back and correct the data. It will disappear from the database in about a month.
• # of New Connections per Second by Hour Frequency
• This just helps to visualize where the systems averages are at. It helps show that most hours will be less than 30 connections per second; which is less than 2 LCU. (1 LCU = 25 new connections per second)

Next Up, AWS ALB Price Planning w/ IIS : Active Connections.