Excessive server requests from vmsACARS

[nicky9499]

Hi gents,

 

I just had an episode whereby my web host disconnected all my sites due to "excessive server requests".

While this was resolved after much grief with their support staff, the core issue is not solved, and this is also not the first time such an event has happened.

 

This time, the support agent kindly provided full logs for the 15 minutes leading up to the disconnection. https://pastebin.com/tLHyNS0j

Naturally, I don't know what these mean at all but the server was indeed being hit with an alarming amount of consecutive requests.

 

Examples:

"POST /api/pireps/2Axjw96wnZOAVrlm/acars/positions HTTP/1.1" 200 41 "-" "RestSharp/106.12.0.0"

"POST /api/pireps/vYJm58W54j9VmJXG/update HTTP/1.1" 200 97599 "-" "RestSharp/106.12.0.0"

"POST /api/pireps/vYJm58W54j9VmJXG/acars/logs HTTP/1.1" 200 36 "-" "RestSharp/106.12.0.0"

"POST /api/pireps/2Axjw96wnZOAVrlm/update HTTP/1.1" 200 107528 "-" "RestSharp/106.12.0.0"

"POST /api/pireps/3Gj3MxNMYL7nP1RN/fields HTTP/1.1" 200 134 "-" "RestSharp/106.12.0.0"

 

2186 requests over a period of 15 minutes. Is this normal?

I still have no idea what an "acceptable" rate is and there is no clarity from the host on this as well.

My cPanel shows that I have plenty of resources left https://prnt.sc/22ris03

 

In the interim, I have doubled all refresh timings from 15 to 30 seconds in vmsacars settings.

However I still don't know if this is sufficient or whether or not it will prevent another future outage.

 

Can someone advise please?

 

[Nabeel]

How many flights were going on? Do you have timestamps with the logs? 30 seconds is still pretty frequent.

[nicky9499]

G'day Nabeel,

 

Thanks for the reply. There were 10+ flights in the air at that time, with another 1 or 2 staff poking around in the backend (phpvms admin).

I'm not sure that is enough to cause an outage, because we certainly had a higher level of activity in the past on smartCARS.

This is the second such outage we have had since switching to vmsACARS, hence my concern.

 

The full log is here https://pastebin.com/tLHyNS0j

Timestamps are in UTC +1.

 

Edit:

In follow-up correspondence, an agent sent me this screenshot of my account, citing "high usage" but not providing any more details.

 

Is it the "hits" that are abnormally high? If I check awstats in cPanel, the highest hit pages are all acars-related:

https://prnt.sc/22rlc9a

Edited December 12, 2021 by nicky9499

[Nabeel]

That sounds normal. Reduce your data collection rate, I don't think it needs to be that high. They're just talking about request rate. Every 2 minutes is probably a good amount.

 

Otherwise you probably need to look into a VPS, you might just be outgrowing shared hosting

 

I'll look at the code and see if there's anything I missed, though

[DisposableHero]

Technically speaking, it all depends on our (va admins) settings and the hosting service provider limits.

 

15 and 30 seconds are too low and will generate lots of requests (+db hits/entries) for nothing. By "nothing" I mean they have no practical use unless you are checking every single pirep for taxi routes in detail, or flight routes over the map etc. And this logic applies to all phases of the flight (regarding vmsAcars module settings).

 

Ground : 60 seconds (maybe 30 if you want nicer looking taxi paths on the map or 15 seconds if you are that hardcore and using that data for something)

Airborne below 18k : 60 seconds or higher (again if you plan to follow SID/STAR execution deeply and plan to play with that data, then 30 seconds)

Airborne above 18k : 120 seconds or higher (practically this is cruise phase, so only map position will change and the map will look better when reduced) 

 

I saw some virtual airlines setting every one of these to 15 seconds, then complaining about server performance, then again those va's was not even using the data at all. No map checks, no special code running in the background reading that data huge data they pulled, simply nothing, probably they were just thinking that "smaller the time, the better it works". In fact, setting those times will not change how or what vmsAcars will record and report back to phpVms. The core data, the important bits will always be there. So by setting all of the above to 240 seconds (for example), you will not loose the take off speed or landing pitch, fuel figures or a violation etc.

 

So, why it was not happening with other software (like SmartCars) ?

Most probably you did not had that low settings with it or it was not sending the data back frequently. Imagine like you set all to 15 seconds, software records them internally every 15 seconds but sends them to phpvms every 120 seconds to reduce requests and traffic. Packs them into a single requests and sends only it, like downloading a zip file instead of 4 separate files. In the end you will see the data, but not real time (if it matters of course)

 

And why it differs from hosting to hosting ?

It is how they manage their server load and customer happiness, server uptime etc. One may have more strict restrictions, the other may be much relaxed on the same setting. This is the main reason they are offering "cheap" solutions compared to "VPS" or "Dedicated Server" 'cause they are in control. There may be hundreds more websites (customers) on that same server (with unlimited bandwidth, unlimited disk space, unlimited database rights etc) , when you start to use more "main" resources you will be the "one" to blame and your traffic will be reduced (or blocked) to save the remaining %99.

 

What are the solutions;

 

1. Reduce the traffic/requests by adjusting the times to more sensible values

2. Consider changing hosting service provider

3. Consider switching to a different package (like using VPS, in which you will control the server and it will not be like using a "shared" hosting)

 

Safe flights

[nicky9499]

The server has just gone down again, this time my host has suspended the account on grounds of being under a DDoS attack.

There was only 1 flight in the air at that time. This is the relevant log when the "DDos" occurred.

https://paste.websitehostserver.net/text_2021-12-12_15-52-05_SeuPf7O1Yd

 

I am fighting with the hosting agents to regain access and further raise these update intervals, but I am beginning to feel there might be something else causing this especially since only 1 pilot was flying in the latest incident, which doesn't make any sense whatsoever.

Edited December 13, 2021 by nicky9499

[DisposableHero]

Do not share somebody else's (or your own) API key publicly please. It is important for security.

 

I highly advise that user to generate a new api key once you regain access to your site/hosting account.

Edited December 12, 2021 by DisposableHero

[Nabeel]

15 hours ago, nicky9499 said:

The server has just gone down again, this time my host has suspended the account on grounds of being under a DDoS attack.

There was only 1 flight in the air at that time. This is the relevant log when the "DDos" occurred.

https://paste.websitehostserver.net/text_2021-12-12_15-52-05_SeuPf7O1Yd

 

I am fighting with the hosting agents to regain access and further raise these update intervals, but I am beginning to feel there might be something else causing this especially since only 1 pilot was flying in the latest incident, which doesn't make any sense whatsoever.

 

What's your ACARS update timer set to? Because that looks like it's updating every few seconds. But let me look at that code... it could be a bug.

[Nabeel]

I think I've found an issue with this, where it's not properly reading it from the remote ACARS config setting ("ACARS Update Timer"). I'm pushing a fix to the beta channel right now.

[nicky9499]

Hi Nabeel,

 

Sorry for the delayed reply. Here are my update interval settings:

 

Thank you for your work on diagnosing and fixing the issue. What steps should I advise the pilots to take to ensure that they are on this "beta channel"?

[Nabeel]

Download the latest phpvms dev version. There's an option in the ACARS settings for the beta.