Best practices to connect thirdparty services from ATG

This issue typically occurs when ATG uses a client to access external hosted services over http when the read/connection timeout are not configured.The speed and integrity of the external services is outside of the control of ATG however if these services are unavailable any attempt to connect to them over http will result in a connection (or thread) which does not timeout (the default http connection timeout is configured to -1 or 0 which is infinite). This results in a build of hanging threads in the ATG application which wait on the default http timeout..

Recommendations

Below is a list of recommendations to overcome outages

1.    Prevent client requests invoking application threads indefinitely. Set the sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout

2.    Prevent application from using an thirdparty services which are unavailable. Implement a mechanism for preventing unnecessary requests to a service which is unavailable.

3.    Ephemeral ports within ATG configuration should not be used. The ephemeral port range in linux should not be used by ATG configurations as this can result in port conflicts on startup.

4.    Use any third party services alternate endpoints when primary host is down.

Prevent client requests invoking application threads indefinitely

The two JVM arguments sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout are set on the JVM and It is recommended that both of these arguments are set to 10000 ms (or 10 seconds).

It is note that while setting these values will help prevent clients from tying up threads indefinitely, it will not prevent multiple threads from being instantiated, and an outage can still happen under heavy load if there are a lot of threads waiting 10 seconds.

Prevent application from using an thirdparty services which are unavailable

To prevent application from creating unnecessary requests to third party services which are not available or down .Then following approach should be used:

Using this approach, will ensure that ATG is not overwhelmed by blocking requests as the first request to fail will set the service to unavailable to all other clients which use the service.  When the service resumes a scheduled services will mark it as available again allowing clients to use it again.

Ephemeral ports within ATG configuration should not be used

When a client makes a TCP request e.g. when calling third party service(ex: vertex tax or payment gateway), the request uses the servers local ip address and a port number to establish the communication channel. When a port is not specifically assigned to the client request, a port is allocated from the ephemeral port range.  Ephemeral ports are temporary allocated for each request and this range is used specifically for this purpose.  Therefore applications which have fixed port allocations should not be configured to use ports in the ephemeral port range, as these ports might not be available and prevent ATG from binding to a particular port.

The default  ephemeral port range on the Linux kernel is 32768 through 61000 if adequate kernel memory is available, meaning that no ports should be used by ATG in this range.

Use thirdparty alternate endpoints when primary is host is down

Most of the thirdparty services are exposed with redundant hostname/port network endpoints to ensure high availability for its clients. For more availability, Developers should include code to find connectivity issues and HTTP errors, and temporarily switch to a alternate URL. Alternate secondary hostname/port must be automatic and completely transparent to the end-user. Communication with the primary hostname/port should be attempted periodically while connecting to alternate endpoint.