IIS Proxy & App Web Performance Optimizations Pt. 4

Last time we took the new architecture to it’s theoretical limit and pushed more of the load toward the database. This time …

What we changed (Using 2 Load Test Suites)

   
• Turn on Output Caching on the Proxy Server. Which defaults to caching js, css, and images. Which works really great with really old sites.
• We also lowered the number of users as the Backend Services ramped up to 100%.
• Forced Test Agents to Run in 64-bit mode. This resolved the an Out Of Memory exception that we we’re getting when the Test Agents were running into the 2 GB memory caps of their 32-bit processes.
• Found a problem with the Test Suite that was allowing all tests to complete without hitting the backend service. (This really effected the number of calls that made it to the Impacted Backend Services.)
• Added a second Test Suite which also used the same database. The load on this suite wasn’t very high; it just added more real world requests.

Test Setup

• Constant Load Pattern
• 1000 users
• 7 Test Agents (64-bit mode)
• Main Proxy
• 4 vCPU / 8 vCore
• 24 GB RAM
• AppPool Queue Length: 50,000
• WebFarm Request Timeout: 120 seconds
• Output Caching (js, css, images)
• Impacted Web App Server
• 3 VMs
• AppPool Queue Length: 50,000
• Impacted Backend Service Server
• 8 VMs
• Classic ASP App
• CDNs used for 4 JS files and 1 CSS file
• Custom JS and CSS coming from Impacted Web App
• Images still coming from Impacted Web App
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Enabled
• A 2nd Test Suite which Impacts other applications in the environment is also run. (This is done off a different VS 2017 Test Controller)

Test Results

• Main Proxy
• CPU: 28% (down 37)
• Max Concurrent Connections: – (Didn’t Record)      
• Impacted Web App
• CPU: 56% (down 10)
• Impacted Backend Service
• CPU: 100% (up 50)
• DB
• CPU: 30% (down 20)
• VS 2017 Test Suite
• Total Tests: 95,000 (down 30,000)
• Tests/Sec: 869 (down 278)

This more “real world” test really highlighted that the impacted systems weren’t going to have a huge impact on the database shared by the other systems which will using it at the same time.

We had successfully moved the load from the Main Proxy onto the the backend services, but not all the way to the database. With some further testing we found that adding CPUs and new VMs to the Impacted Backend Servers had a direct 1:1 relationship with handling more requests. The unfortunate side of that is that we weren’t comfortable with the cost of the CPUs compared to the increased performance.

The real big surprise was the significant CPU utilization decrease that came from turning On Output Caching on the

And, with that good news, we called it a day.

So, the final architecture looks like this …

What we learned …

• SSL Encryption/Decryption can put a significant load on your main proxy/load balancer server. The number of requests processed by that server will directly scale into CPU utilization. You can reduce this load by moving static content to CDNs.
• Even if your main proxy/load balancer does SSL offloading and requests to the backend services aren’t SSL encrypted, the extra socket connections still have an impact on the servers CPU utilization. You can lower this impact on both the main proxy and the Impacted Web App servers by using Output Caching for static content (js, css, images).
• We didn’t have the need to use bundling and we didn’t have the ability to do spriting; but we would strongly encourage anyone to use those if they are an option.
• Moving backend service requests to an internal proxy doesn’t significantly lower the number of requests through the main proxy. It’s really images that create the most number of requests to render a web page (especially with an older Classic ASP site).
• In Visual Studio, double check that your suite of web tests are doing exactly what you think they are doing. Also, go the extra step and check that the HTTP Status Code returned on each request is the code that you expect. If you expect a 302, check that it’s a 302 instead of considering a 200 to be satisfactory.

IIS Proxy & App Web Performance Optimizations Pt. 3

We left off last time after resolving 3rd party JS and CSS files from https://cdnjs.com/ CDNs. And having raised the Main Proxy servers Application Pool Queue Length from 1,000 to 50,000.

We are about to add more CPUs to the Main Proxy and see if that improves throughput.

What we changed (Add CPUs)

• Double the number of CPUs to 4 vCPU / 8 vCore.
• So far the number of connections into the proxy directly correlates to the amount of cpu utilization / load. Hopefully, by adding more processing power, we can scale up the number Test Agents and the overall load.

Test Setup

• Step Load Pattern
• 1000 initial users, 200 users every 10 seconds, max 4000 users
• 4 Test Agents
• Main Proxy
• 4 vCPU / 8 vCore
• 24 GB RAM
• AppPool Queue Length: 50,000 (default)
• WebFarm Request Timeout: 30 seconds (default)
• Impacted Web App Server
• 2 VMs
• Impacted Backend Service Server
• 6 VMs
• Classic ASP App
• CDNs used for 4 JS files and 1 CSS file
• Custom JS and CSS coming from Impacted Web App
• Images still coming from Impacted Web App
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Enabled

Test Results

• Main Proxy
• CPU: 65% (down 27)
• Max Concurrent Connections: 15,000 (down 2,500)
• Impacted Web App
• CPU: 87%
• Impacted Backend Service
• CPU: 75%
• VS 2017 Test Suite
• Total Tests: 87,000 (up 22,000)
• Tests/Sec: 794 (up 200)

Adding the processing power seemed to help out everything. The extra processors allowed for more requests to be processed in parallel. This allowed for requests to be passed through and completely quicker, lower the number of concurrent requests. With the increased throughput the number of Tests that could be completed, increases the number of Tests/Sec.

Adding more CPUs to the Proxy helps everything in the system move faster. It parallelizes the requests flowing through it and prevents process contention.

So, where does the new bottleneck exist?

Now that the requests are making it to the Impact Web App, the CPU load has transferred to them and their associated Impacted Backend Services. This is a good thing. We’re moving the load further down the stack. Doing that successfully would push the load down to the database (DB); which is currently not under much load at all.

What we changed (Add more VMs)

• Added 1 more Impacted Web App Server
• Added 2 more Impacted Backend Services Servers
• The goal with these additions to use parallelization to allow for more requests to be processed at once and push the bottleneck towards the database.

Test Setup

• Step Load Pattern
• 1000 initial users, 200 users every 10 seconds, max 4000 users
• 4 Test Agents
• Main Proxy
• 4 vCPU / 8 vCore
• 24 GB RAM
• AppPool Queue Length: 50,000 (default)
• WebFarm Request Timeout: 30 seconds (default)
• Impacted Web App Server
• 3 VMs
• Impacted Backend Service Server
• 8 VMs
• Classic ASP App
• CDNs used for 4 JS files and 1 CSS file
• Custom JS and CSS coming from Impacted Web App
• Images still coming from Impacted Web App
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Enabled

Test Results

• Main Proxy
• CPU: 62% (~ the same)
• Max Concurrent Connections: 14,000 (down 1,000)
• Impacted Web App
• CPU: 60%
• Impacted Backend Service
• CPU: 65%
• VS 2017 Test Suite
• Total Tests: 95,000 (up 8,000)
• Tests/Sec: 794 (~ the same)

The extra servers helped get requests through the system faster. So, the overall number of Tests that completed increased. This helped push the load a little further down.

The Cloud philosophy of handling more load simultaneously through parallelization works. Obvious, right?

So, in that iteration, there was no bottleneck. And, we are hitting numbers similar to what we expect of the day of the event. But, what we really need to do leave ourselves some head room in case more users show up that we expect. So, let’s add in more Test Agents and see what it can really handle.

What we changed (More Users Than We Expect)

• Added more Test Agents in order to overload the system.

Test Setup

• Step Load Pattern
• 2000 initial users, 200 users every 10 seconds, max 4000 users
• 7 Test Agents
• Main Proxy
• 4 vCPU / 8 vCore
• 24 GB RAM
• AppPool Queue Length: 50,000
• WebFarm Request Timeout: 30 seconds (default)
• Impacted Web App Server
• 3 VMs
• Impacted Backend Service Server
• 8 VMs
• Classic ASP App
• CDNs used for 4 JS files and 1 CSS file
• Custom JS and CSS coming from Impacted Web App
• Images still coming from Impacted Web App
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Enabled

Test Results

• Main Proxy
• CPU: 65% (~ same)
• Max Concurrent Connections: 18,000 (up 4,000)
• Impacted Web App
• CPU: 63%
• Impacted Backend Service
• CPU: 54%
• VS 2017 Test Suite
• Total Tests: 125,000 (up 30,000)
• Tests/Sec: 1147 (up 282)

So, the “isolated environment” limit is pretty solid but we noticed that at these limits the response time on the requests had slowed down in the beginning of the Test iteration.

.asp Page Response Times

The theory is that with 7 Test Agents, all of which started out 2,000 initial users with no caches primed, all made requests for js, css, and images which swamped the Main Proxy and the Impacted Web App servers. Once the caches started being used in the tests, then things started to smooth out and things stabilized.

From this test we found two error messages started occurring on the proxy. The first error was 502.3 Gateway Timeout and 503 Service Unavailable. Looking at the IIS logs on the Impacted Web App server we could see that many requests (both 200 and 500 return status codes) were resolving with a Win32 Status Code of 64.

To resolve the Proxy 502.3 and then Impacted Web App Win32 Status Code 64 problems we increased the Web Farm Request Timeout to 120 seconds. This isn’t ideal, but from what you can see in the graphic above, the average response time is consistently quick. So, this will ensure all users will get a response, even though some may have a severely degraded experience. Chances are, their next request will process quickly.

Happily, the 503 Service Unavailable was not being generated on the Main Proxy server. It was actually being generated on the Impact Web App servers. They still had their Application Pool Queue Length set to the default 1,000 requests. We increased those to 50,000 and that removed that problem.

Next Time …

We’ll add another Test Suite to run along side it and look into more Caching.

IIS Proxy & App Web Performance Optimizations Pt. 2

Continuing from where we left off in IIS Proxy & App Web Performance Optimizations Pt. 1, we’re now ready to run some initial tests and get some performance baselines.

The goal of each test iteration is to attempt to load the systems to a point that a bottleneck occurs and then find how to relieve that bottleneck.

Initial Test Setup

• Step Load Pattern
• 100 initial users, 20 users every 10 seconds, max 400 users
• 1 Test Agent

Initial Test Results

There was no data really worth noting on this run as we found the addition of the second proxy server lowered the overhead on the Main Proxy enough that no systems were a bottleneck at this point. So, we added more Test Agents re-ran the test with:

Real Baseline Test Setup

• Step Load Pattern
• 1000 initial users, 200 users every 10 seconds, max 4000 users
• 3 Test Agents
• Main Proxy
• 2 vCPU / 4 vCore
• 24 GB RAM
• AppPool Queue Length: 1000 (default)
• WebFarm Request Timeout: 30 seconds (default)
• Impacted Web App Server
• 2 VMs
• Impacted Backend Service Server
• 6 VMs
• Classic ASP App
• No CDNs used for JS, CSS, or images
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Disabled

Real Baseline Test Results

• Main Proxy
• CPU: 99%
• Max Concurrent Connections: 17,000
• VS 2017 Test Suite
• Total Tests: 37,000
• Tests/Sec: 340

In this test we discovered that around 14,000 connections was the limit of the Main Proxy before we started to receive responses on 503 Service Unavailable. We didn’t yet understand that there was more to it, but we set about trying to lower the number of connections by lowering the number of requests for js, css, and images. Looking through the IIS logs we also saw the majority of requests were for the static content; which made it look like information wasn’t being cached between calls. So, we found a setting in VS 2017’s Web Test that allowed us to enable caching. (We also saw a lot of the SocketExceptions mentioned in the previous post, but we didn’t understand what they meant at that time).

What we changed (CDNs and Browser Caching)

• We took all of the 3rd party JS and CSS files that we use and referenced them from https://cdnjs.com/ CDNs. In total, there was 4 js files and 1 css file.
• The reason this hadn’t been done before is there wasn’t enough time to test the fallback strategies if the CDN doesn’t serve the js/css, then the browser should request the files from our servers. We implemented these fallbacks this time.
• We updated the VS 2017 Web Test configuration to enable caching. Whenever a new Test scenario is run, the test agent will not have caching enabled in order to replicate a “new user” experience; each subsequent call in the scenario will use cached js, css, and images. (This cut around 50% of the requests made in the baseline test)
• The majority of the requests into the Main Proxy were image requests. But, the way the application was written we couldn’t risk a) moving the images to a CDN or b) spriting the images. (It is a Classic ASP app, so it doesn’t have all the bells and whistles that newer frameworks have)

Test Setup

• Step Load Pattern
• 1000 initial users, 200 users every 10 seconds, max 4000 users
• 3 Test Agents
• Main Proxy
• 2 vCPU / 4 vCore
• 24 GB RAM
• AppPool Queue Length: 1000 (default)
• WebFarm Request Timeout: 30 seconds (default)
• Impacted Web App Server
• 2 VMs
• Impacted Backend Service Server
• 6 VMs
• Classic ASP App
• CDNs used for 4 JS files and 1 CSS file
• Custom JS and CSS coming from Impacted Web App
• Images still coming from Impacted Web App
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Enabled

Test Results

• Main Proxy
• CPU: 82% (down 17)
• Max Concurrent Connections: 10,400 (down 6,600)
• VS 2017 Test Suite
• Total Tests: 69,000 (up 32,000)
• Tests/Sec: 631 (up 289, but with 21% failure rate)

Offloading the common third party js and css files really lowered the number of requests into the Main Proxy server (38% lower). And, with that overhead removed, the CPU utilization came down from a pegged 99% to 82%.

Because caching was also enabled, the test suite was able to churn through the follow-up page requests much quicker. That increase in rate nearly doubled the number of Tests/Sec completed.

Move 3rd party static content to CDNs when possible (https://cdnjs.com/ is a great service.) When doing so, try to implement failed loads and fallbacks for those resources.

But, we still had high CPU utilization on the Main Proxy. And, we had a pretty high failure rate with lots of 503 Service Unavailable and some 502.3 Gateway Timeouts. We determined the cause of the 503s was that the Application Pools Queue length was being hit. We considered this to be the new bottleneck.

What we changed (CDNs and Browser Caching)

• We set the application pool queue length from 1,000 to 50,000. This would allow us to queue up more requests and lower the 503 Service Unavailable error rate.
• We also had enough head room in the CPU to add another Test Agent.

Test Setup

• Step Load Pattern
• 1000 initial users, 200 users every 10 seconds, max 4000 users
• 4 Test Agents
• Main Proxy
• 2 vCPU / 4 vCore
• 24 GB RAM
• AppPool Queue Length: 50,000
• WebFarm Request Timeout: 30 seconds (default)
• Impacted Web App Server
• 2 VMs
• Impacted Backend Service Server
• 6 VMs
• Classic ASP App
• CDNs used for 4 JS files and 1 CSS file
• Custom JS and CSS coming from Impacted Web App
• Images still coming from Impacted Web App
• JS is minified
• VS 2017 Test Suite
• WebTest Caching Enabled

Test Results

• Main Proxy
• CPU: 92% (up 10)
• Max Concurrent Connections: 17,500 (up 7,100)
• VS 2017 Test Suite
• Total Tests: 65,000 (down 4,000)
• Tests/Sec: 594 (down 37, but with only 3% failure rate)

This helped fix the failure rate issue. Without all the 503s forcing the Tests to end early, it took slightly longer to complete each test and that caused the number of Tests/Sec to fall a bit. This also meant we had more requests queued up, bringing the number of concurrent connections back up.

For heavily trafficked sites, set your Application Pool Queue Length well above the default 1,000 requests. This is only needed if you don’t have a Network Load Balancer in front of your proxy.

At this point we were very curious what would happen if we added more processors to the Main Proxy. We were also curious what the average response time was from the Classic .asp pages. (NOTE: all the js, css, and image response times are higher than the page result time.)

.asp Page Response Times on Proxy

Next Time …

We’ll add more CPUs to the proxy and see if we can’t push the bottleneck further down the line.

IIS Proxy & App Web Performance Optimizations Pt. 1

We’re ramping up towards a day where our web farm fields around 40 times the normal load. It’s not much load compared to truly popular websites, but it’s a lot more than what we normally deal with. It’s somewhere around the order of ~50,000 people trying to use the system in an hour. And, the majority of the users hit the system in the first 15 minutes of the hour.

So, of course, we tried to simulate more than the expected load in our test environment and see what sort of changes we can make to ensure stability and responsiveness.

A quick note: This won’t be very applicable to Azure/Cloud based infrastructure. A lot of this will be done for you on the Cloud.

Web Farm Architecture

These systems run in a private Data Center. So, the servers and software don’t have a lot of the very cool features that the cloud offers.

The servers are all Win 2012 R2, IIS 8.5 with ARR 3.0, URL Rewrite 7.2, and Web Farm Framework 1.1.

Normally, the layout of the systems is similar to this diagram. This gives a general idea that there is a front-end proxy, a number of applications, backend services, and a database which are all involved in this yearly event. And, that a single Web App is significantly hit and it’s main supporting Backend Service is also significantly hit. The Backend Service is also shared by the other Web Apps involved in the event; but they are not the main clients during that hour.

Testing Setup

For testing we are using Visual Studio 2017 with a Test Controller and several Agents. It’s a very simple web test suite with a single scenario. This is the main use case during that hour. A user logs in to check their status, and then may take a few actions on other web applications.

Starting Test Load

• Step Pattern
• 100 users, 10 user step every 10 seconds, max 400 users
• 1 Agent

We eventually get to this Test Load

• Step Pattern
• 1000 users, 200 user step every 10 seconds, max 2500 users
• 7 agents

We found that over 2500 concurrent users would result in a SocketException on the Agent machines. Our belief is that each agent attempts to run the max user load defined by the test. And, that the Agent Process will run out (Sockets?) to spawn new users to make calls. This results in SocketExceptions. To alleviate the issue, we added more Agents to the Controller and lowered the maximum number of concurrent users.

SocketExceptions on VS 2017 Test Agents can be prevented by lowering the maximum number of concurrent users. (You can then add in more Agents to the Test Controller in order to get the numbers back up.)

Initial Architecture Change

We’ve been through this load for many years so we already have some standard approaches that we take every year to help with the load:

• Add more Impacted Backend Service servers
• Add more CPU/Memory to the Impacted Web App

This year we went further by

• Adding another proxy server to ensure Backend Service Calls from the Impacted Web App don’t route through the Main Proxy to the Impacted Backend Services. This helps reduce the number of connections through the Main Proxy.
• Adding 6 more Impacted Backend Service servers. These are the servers that always take the worst hit. These servers don’t need sticky sessions, so they can easily spread the load between them.
• Adding a second Impacted Web App server. This server usually doesn’t have the same level of high CPU load that the Proxy and Impacted Backend Services do. These servers do require sticky sessions, so there are potential issues with the load not being balanced.

If you don’t have to worry about sticky session, adding more processing servers can always help distribute a load. That’s why Cloud based services with “Sliders” are fantastic!

Next Time …

In the next section we’ll look at the initial testing results and the lessons learned on each testing iteration.

Apigee Response CORS Headers using Javascript

Apigee provides a quick “Add CORS Headers” to responses when creating a new API Proxy. It’s straight forward and will get you started to add CORS headers to the replies from your first API endpoints. The problem with that is that CORS headers are used in “preflight” and aren’t that useful after the call has successfully completed. Apigee OPTIONS Response for Preflight/CORS can help you set up preflight responses.

But, it’s still useful to add in CORS headers to your responses in order to ensure that your endpoints are communicating their security requirements. To do this you can use javascript to inspect the responses and add in missing CORS headers. This sample javascript will:

• Ensure Access-Control-Allow-Origin is defined. Sets the default value to ‘*’.
• Ensure Access-Control-Allow-Headers is defined. Sets the default value to ‘origin, x-requested-with, accept, my-api-key, my-api-version, authorization, content-type’.
  • my-api-key and my-api-version are custom headers specific to the Apigee endpoints this script is used with. If the Resource Service doesn’t return these headers, then they will be added in.
• Ensure Access-Control-Max-Age is defined. Sets the default value to ‘3628800’ seconds (42 days … I have no idea why that was chosen.)
• Ensure Access-Control-Allow-Methods is defined. Sets the default value to ‘GET, PUT, POST, DELETE’. This should really be set by the Resource Service, so use it only if you feel comfortable.

This should be created as a Shared Flow and applied to Proxy Endpoint's Postflow.

//  Access-Control-Allow-Origin
var accessControlAllowOrigin = context.getVariable("response.header.Access-Control-Allow-Origin.values").toString();
if(accessControlAllowOrigin.startsWith('[')) { accessControlAllowOrigin = accessControlAllowOrigin.substring(1, accessControlAllowOrigin.length() - 1); }
if(accessControlAllowOrigin.endsWith('[')) { accessControlAllowOrigin = accessControlAllowOrigin.substring(0, accessControlAllowOrigin.length() - 1); }
if(accessControlAllowOrigin.length() === 0) {
    accessControlAllowOrigin = "*";
}
context.setVariable("response.header.Access-Control-Allow-Origin", accessControlAllowOrigin);

//  Access-Control-Allow-Headers
var accessControlAllowHeaders = context.getVariable("response.header.Access-Control-Allow-Headers.values").toString();
if(accessControlAllowHeaders.startsWith('[')) { accessControlAllowHeaders = accessControlAllowHeaders.substring(1, accessControlAllowHeaders.length() - 1); }
if(accessControlAllowHeaders.endsWith('[')) { accessControlAllowHeaders = accessControlAllowHeaders.substring(0, accessControlAllowHeaders.length() - 1); }
if(accessControlAllowHeaders.length() === 0) {
    accessControlAllowHeaders = "origin, x-requested-with, accept, my-api-key, my-api-version, authorization, content-type";
}
if(accessControlAllowHeaders.indexOf("my-api-key") === -1) {
    accessControlAllowHeaders += ", my-api-key";
}
if(accessControlAllowHeaders.indexOf("my-api-version") === -1) {
    accessControlAllowHeaders += ", my-api-version";
}
context.setVariable("response.header.Access-Control-Allow-Headers", accessControlAllowHeaders);

//  Access-Control-Max-Age
var accessControlMaxAge = context.getVariable("response.header.Access-Control-Max-Age.values").toString();
if(accessControlMaxAge.startsWith('[')) { accessControlMaxAge = accessControlMaxAge.substring(1, accessControlMaxAge.length() - 1); }
if(accessControlMaxAge.endsWith('[')) { accessControlMaxAge = accessControlMaxAge.substring(0, accessControlMaxAge.length() - 1); }
if(accessControlMaxAge.length() === 0) {
    accessControlMaxAge = "3628800";
}
context.setVariable("response.header.Access-Control-Max-Age", accessControlMaxAge);

//  Access-Control-Allow-Methods
var accessControlAllowMethods = context.getVariable("response.header.Access-Control-Allow-Methods.values").toString();
if(accessControlAllowMethods.startsWith('[')) { accessControlAllowMethods = accessControlAllowMethods.substring(1, accessControlAllowMethods.length() - 1); }
if(accessControlAllowMethods.endsWith('[')) { accessControlAllowMethods = accessControlAllowMethods.substring(0, accessControlAllowMethods.length() - 1); }
if(accessControlAllowMethods.length() === 0) {
    accessControlAllowMethods = "GET, PUT, POST, DELETE";
}
context.setVariable("response.header.Access-Control-Allow-Methods", accessControlAllowMethods);

Apigee Key Value Maps (KVM) To Store Passwords

Cloud based computing has broken some of the molds of traditional security models. Things like IP whitelisting on a firewall sometimes aren’t even an option. And, because of that, some older techniques are back and can really work wonders for simple authentication security.

A quick note: Apigee’s Business option (and above) actually comes with static IP addresses, so this technique can be used in conjunction with IP whitelisting.

Apigee is an API Management system, so it has the capability to handle many authentication protocols for clients to connect to it’s cloud based endpoints. But, we’re gonna look at the other half to the communication path. We’re gonna look at when the API Gateway has to call down to the resource service. And, this is a technique to inform the resource server that it is the API Gateway which is making the call to it.

The technique is Basic Authentication. It’s been around for a long time and it’s basically a magic string you put in the header of your requests. Your resource service will inspect the header and make sure it’s talking to a client that knows the shared secret. Since this is a shared secret we need a way to store the secret in Apigee that’s secure. And, it’s pretty darn secure.

Put the Shared Secret in the KVM

Basic Authentication is a username and password joined together by a colon and then base 64 encoded. The header looks like this:

Authorization:    Basic   {base64encoded(“username:password”)}

So, we’re going to store both the username and password into Apigee’s KVM. The first thing we need to do is select the KVM level we want to store it at.

• Organization Level
• If you’re going to reuse the same username/password on multiple APIs in multiple environments, then this works well.
• Environment Level
• If you’re going to reuse the same username/password on multiple APIs, but you want to use a different secret between Prod and everything else.
• API Proxy Level
• If you’re looking for a secret defined to a single API Proxy, but used in all environments.

In this example, were going to do an API Proxy Level KVM.

$adminUser = "tom@place.com"   # apigee.com/edge username
$adminPass = "tommyspass"      # apigee.com/edge password
$org = "org1"                  # apigee.com/edge organization
$apiName = "my-api"            # an api proxy name

# bump up TLS to 1.2 (.NET defaults to SSL3, which isn't supported on Apigee management endpoints)
[System.Net.ServicePointManager]::SecurityProtocol = 
				[System.Net.SecurityProtocolType]::Tls12 + [System.Net.SecurityProtocolType]::Tls11 + [System.Net.SecurityProtocolType]::Tls;

# this isn't the KVM, this is Apigee security
$bytes = [System.Text.Encoding]::ASCII.GetBytes($adminUser + ":" + $adminPass)
$encodedText = [Convert]::ToBase64String($bytes)
$adminHeader = @{ Authorization = "Basic $encodedText"; "Content-Type" = "application/json" }

$rootUrl = "https://api.enterprise.apigee.com/v1/organizations/$org"
$apikvmUrl = "$rootUrl/apis/$apiName/keyvaluemaps"

$kvms = Invoke-RestMethod -Method GET -Uri $apikvmUrl -Headers $adminHeader
# currently $kvms is most likely empty

# so, let's add one
$kvmName = "my-customKVM"
$kvmEntry = @{
    name = $kvmName
    encrypted = "true"    # this is important and will come back later
    entry = @(
        @{ name = "username"; value = "sooo" },
        @{ name = "password"; value = "secret" }
    )
}
$json = ConvertTo-Json $kvmEntry

$newKvm = Invoke-RestMethod -Method POST -Uri $apikvmUrl -Headers $adminHeader -Body $json
$newKvm | fl
# this is the last time you can see the unencrypted secret values
# make sure to store the values in a password safe before clearing these values

$kvms = Invoke-RestMethod -Method GET -Uri $apikvmUrl -Headers $adminHeader
$kvms # $kvms should now list "new-customKVM"

$apiKvmEntryUrl = "$apikvmUrl/$kvmName"
$kvm = Invoke-RestMethod -Method GET -Uri $apiKvmEntryUrl -Headers $adminHeader
$kvm | fl
# this time the values are hidden (******)

## And, of course the delete
#Invoke-RestMethod -Method Delete -Uri $apiKvmEntryUrl -Headers $adminHeader

Retrieve the Shared Secret from the KVM

Now we have my-customKVM setup at the API Proxy level. So, let’s use the value in the flow to create a Basic Authorization header and populate the value. To do this, we are going to use the KeyValueMapOperations Policy to retrieve the credentials. In this policy, it’s very important to store the credentials to a variable that starts with private.. The KVM entry that we made was encrypted. And, you can only read an encrypted KVM value into a variable that is scoped to private.. The reason for this is security. private. variables will never appear in the Trace tool, nor will they be logged. However, you can look at them by using javascript callouts (this is important for debugging).

You can only read an encrypted KVM value into a variable that is scoped to private.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<KeyValueMapOperations async="false" continueOnError="false" enabled="true" name="Retrieve-Credentials" mapIdentifier="my-customKVM">
    <DisplayName>Retrieve Credentials</DisplayName>
    <Properties/>
    <ExclusiveCache>false</ExclusiveCache>
    <ExpiryTimeInSecs>300</ExpiryTimeInSecs>
    <Scope>apiproxy</Scope>
    <Get assignTo="private.ba.username" index="1">
        <Key>
            <Parameter>username</Parameter>
        </Key>
    </Get>
    <Get assignTo="private.ba.password" index="1">
        <Key>
            <Parameter>password</Parameter>
        </Key>
    </Get>
</KeyValueMapOperations>

And, Assign the Header to the Request

So, we’ve now loaded the username and password from the KVM into private.ba.username and private.ba.password. We are now going to use the BasicAuthentication Policy to set the Authorization header.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<BasicAuthentication async="false" continueOnError="false" enabled="true" name="Add-BasicAuth-Header">
    <DisplayName>Add BasicAuth Header</DisplayName>
    <Operation>Encode</Operation>
    <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
    <User ref="private.ba.username"/>
    <Password ref="private.ba.password"/>
    <AssignTo createNew="false">request.header.Authorization</AssignTo>
</BasicAuthentication>

Troubleshooting private. variables

When working with private. variables, it’s very useful to use Javascript policies to inspect the value of the variables (because they don’t appear in the Trace tool).

print(context.getVariable("private.ba.username"))
print(context.getVariable("private.ba.password"))

Alter PathSuffix in Apigee with/out Load Balancer

Apigee’s API Gateway is by many measures a proxy server with some really nice bells and whistles attached. But, it’s still a proxy server at it’s core. Which means it should be able to transform an incoming request before it’s sent to the backend/target resource servers. It can do that, but it’s not as easy as you might hope.

With an API Gateway, a common trasnformation would be to remove a version number from a url before sending the request to the backend server. This scenario crops up when the developer of the resource API didn’t design their system with version numbers in mind. The scenario looks like this:

So, in this scenario, the Body API Proxy has a BasePath of /body (proxy.basepath). And the PathSuffix would be /v1/wheels?drive=4WD (proxy.pathsuffix). The developer of the resource service didn’t have version built into the url path, and is expecting a url without it.

Without a Load Balancer Configuration

To make this transformation, we are going to need to artificial create the target endpoints url during the flow process. Seeing that the API Gateway is a proxy server, you would think that you would just need overwrite the request or proxy variables, but most of those are actually read only. Here’s what you’ll need to do:

1. You’ll use the request.uri and proxy.basepath to figure out the full path suffix.
2. If the path contains a version number (/v1/) then you will …
3. Set target.copy.pathsuffix to false. (At the moment, you have to use a Javascript Callout. There is a bug with using an AssignMessage Policy).
1. This must occur in the Target Endpoint flows (most likely the PreFlow). You can’t do this in the Proxy Endpoint, because the target variables haven’t been created yet. So, they aren’t “in scope”.
4. You’ll then remove the version number (/v1/) to get the “new” path suffix.
5. And, finally, you will set the target.url to constructed path. (target.url is one of the few read/write variables.)

Target Endpoint with No Load Balancer

With a Load Balancer Configuration

Apigee uses a Load Balancer variable in the Target Endpoint configuration to allow for Resource Server DNS hostnames to be dynamic between the environments. Unfortunately, when this is used, the target.url variable is no longer used. And, you need to set target.copy.queryparams to false as well.

To this, you’ll follow the same steps above, but this time you’ll …

5. And, finally, you will set the target.url to constructed path. (target.url is one of the few read/write variables.)
6. Set target.copy.queryparams to false.
7. Set the {newpathsuffix} variable, which will be configured on the Target Endpoint’s Path.

Target Endpoint with Load Balancer

Javascript Callout for Target Endpoint PreFlow: (note that variable {newpathsuffix} isn’t needed when no Load Balancer is involved. It’s being used to make both implementations look similar.)

//  parses the original request to remove the version piece ("/v1", etc)
var basepath = context.getVariable("proxy.basepath")
print("basepath: " + basepath);
var uri = context.getVariable("request.uri");
print("uri: " + uri);
var pathsuffix = uri.substring(basepath.length)
var regex = /(.*)\/v[0-9]+\/(.*)/
var found = regex.exec(pathsuffix)
print("found: " + found)
if(found !== null) {
    //  prevents the request to the backend server from using the original "request.pathSuffix"
    //  this is very important!
    //  the original "request.path" will overwrite whatever we do here if this isn't set
    context.setVariable("target.copy.pathsuffix", false)
    
    // remove the "/v1" part
    var newPathSuffix = found[1]
    if(newPathSuffix.length > 0) { newPathSuffix += "/" }
    newPathSuffix += found[2]
    
    print("newPathSuffix: " + newPathSuffix)
    context.setVariable("newpathsuffix", newPathSuffix)
    
    var targetUrl = context.getVariable("target.url")
    print("target url: " + targetUrl)
    if(targetUrl !== null) {
        
        var pathSuffixRegex = /(.*){newpathsuffix}(.*)/
        var pathSuffixFound = pathSuffixRegex.exec(targetUrl)
        print("pathSuffixFound: " + pathSuffixFound)
        
        if(pathSuffixFound !== null) {
            
            var newUrl = pathSuffixFound[1] + newPathSuffix + pathSuffixFound[2]
            print("new url (replace): " + newUrl)
            context.setVariable("target.url", newUrl);
            
        } else {
            var newUrl = targetUrl + newPathSuffix
            print("new url (append): " + newUrl)
            context.setVariable("target.url", newUrl);
            
        }
    } else {
        // using load balancer
        context.setVariable("target.copy.queryparams", "false") // needed on load balancer
        // the load balancer can use the variable substitution on the  innerText
    }
} else {
    print("newpathsuffix: [empty string]")
    context.setVariable("newpathsuffix", "")
}

target.copy.pathsuffix and target.copy.queryparams

So, these are the key variables that make overwriting the target path possible. The creation of these variables probably has good reasoning behind it, but from an outside perspective they seem really odd. Apigee’s internal system allows you to do a variety of alterations and checks through the Proxy and Target Endpoint flows. These flows can alter most things within the system at the time they execute within the pipeline. BUT, the proxy.pathsuffix and proxy.queryparams are (a) readonly and (b) will overwrite any changes you make to the target.url value. They just ignore everything that happened in the pipeline and override it. This behavior seems to conflict with the way the “flow” system was designed.