问题描述:

On Jan 17th 09:32, one of our services suddenly started throwing 500 errors. It’s an adapter service to a third-party service and we use a HttpClient to make a POST to it (So we make a GET to our service with query string parameters and we transfer this to the third-party app using a POST and parameters in the body). When I post to the third party service manually using postman or curl, it responded fine. So it was a problem with our service. It’s .NET service which uses OWIN middleware, similar to the way .NET core works I think. The problem was that some time ago, the .NET framework was upgraded from 4.5.2 to 4.6 and when doing this in VS, it adds a <httpRuntime targetFramework="4.5.2"/> element to the web.config. This is to make a best effort to preserve the existing behaviour of the app in case there are any breaking changes between framework versions. The person who upgraded didn’t realise and left in the element in the web.config. It worked fine for ages, then suddenly in all environments at the same time (including locally) got busted. I thought it must be something time related in the .NET framework but rolling my system clock back doesn’t fix it! What can I look for, any ideas as to this mystery? Simply upping the web.config to 4.6 fixes it but I have been tasked with investigating it.

Here is the underlying error:

System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

and this is the code, it throws at _client.PostAsync with the above as the InnerException. _client is a System.Net.Http.HttpClient

public async Task<CalculateResponse> Calculate(CalculateRequest request)

{

var env = new RequestEnvelope { Body = { RblsCalculate = request } };

request.LoginId = _username;

request.Password = _password;

var body = XmlConvert.SerializeObject(env);

var content = new StringContent(body, Encoding.UTF8, "application/soap+xml");

var httpResponse = await _client.PostAsync(_endpointPath, content);

var response = XmlConvert.ToObject<ResponseEnvelope>(await httpResponse.Content.ReadAsStreamAsync());

return response?.Body?.RblsCalculateResponse;

}

The third-party did not make any changes, Windows updates did not run (this effected 5 different environments simultaneously). We didn't make any changes. When we deploy, we deploy to a new instance every time, the web.config did not change on the servers and the previous deploy was weeks before.

I have reviewed some of the changes to 4.6 and there are some potentially breaking changes around HttpClient if not using TLSv1.0+ as the protocol, I have checked using Wireshark on one of the servers and we are using TLSv1.2. But it doesn't explain why it suddenly stopped.

Update - Output from trace.log for SSL/TLS tracing as per @Trumpi suggestion

System.Net.Sockets Verbose: 0 : [16292] Data from Socket#52088480::PostCompletion

System.Net.Sockets Verbose: 0 : [16292] 00000000 : 16 03 01 00 88 01 00 00-84 03 01 58 A4 49 35 01 : ...........X.I5.

Update 2 - Removed the unnecessary logs ^^

网友答案:

Interestingly enough, I hit a very similar issue last week (although it was not with .NET Core). I had been calling an API endpoint for months via a daily job and all of a sudden I was getting the same error. It took me several days to find a fix, but for me adding the following line of code fixed the issue. You could probably just add it to the first line of your method.

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
网友答案:

My first instinct is that this is a problem with the TLS handshake and that the third party service is dropping the connection because it could not perform a successful handshake. The TLS version could be an issue, as you noted. Not being able to find a compatible cipher could be another issue.

I stumbled upon this blog post that describes how to write the handshake information out to the trace file. Here is the section that he adds to the web.config file:

<system.diagnostics> <trace autoflush="true"/> <sources> <source name="System.Net" maxdatasize="1024"> <listeners> <add name="TraceFile"/> </listeners> </source> <source name="System.Net.Sockets" maxdatasize="1024"> <listeners> <add name="TraceFile"/> </listeners> </source> </sources> <sharedListeners> <add name="TraceFile" type="System.Diagnostics.TextWriterTraceListener" initializeData="trace.log"/> </sharedListeners> <switches> <add name="System.Net" value="Verbose" /> <add name="System.Net.Sockets" value="Verbose" /> </switches> </system.diagnostics>

This is the best I can do with the information in the question and I hope that this helps.

EDIT: After posting the results, it looks like the call is trying to negotiate a TLS 1.0 connection, which the server no longer supports. I've put the details in a comment below.

相关阅读:
Top