{"id":20345,"date":"2021-02-15T21:45:48","date_gmt":"2021-02-15T21:45:48","guid":{"rendered":"http:\/\/www.devopsschool.com\/blog\/?p=20345"},"modified":"2021-10-28T10:28:34","modified_gmt":"2021-10-28T10:28:34","slug":"ddos-attack-and-http-flood-attack-from-new-relic-synthetics-monitoring-of-aws-ec2-instance","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/ddos-attack-and-http-flood-attack-from-new-relic-synthetics-monitoring-of-aws-ec2-instance\/","title":{"rendered":"DDOS attack and HTTP flood attack on AWS from AWS EC2 instance using New Relic Synthetics & infalted aws bill to $3000"},"content":{"rendered":"\n

How I prevented DDOS attack and HTTP flood attack from New Relic Synthetics Monitoring of AWS EC2 Instance?<\/p><\/blockquote>\n\n\n\n

5 Dec 2020 – DDOS attack and HTTP flood attack Start Date from AWS Ec2 Instance<\/strong><\/p>\n\n\n\n

10 Jan 2021 – I found out this attack when the bill has come for more than $3000.<\/strong><\/p>\n\n\n\n

Phase 1 – Duration 10 Days – Contacting AWS Support team – No Help from them<\/h3>\n\n\n\n

I noticed that my AWS Data transfer bill has come to almost $3000 which is 100 times more than regular. I was shocked and for the moment, I assumed that it was an AWS issue and I reported it to them. After 5-6 days of struggling with the AWS support team who were not able to help me, they asked me to reach out to the AWS ec2 abuse team. Meanwhile everyday, the Data transfer cost was coming almost $100\/day. I was helpless. I started regretting my decision to migrate from Godaddy VPC to an AWS reserved instance. So many hidden charges which you would come to know only once you start using it even if you make a payment in advance for an AWS ec2 reserved instance. Definitely, I felt helpless that AWS is not good for small business with no proper IT team in place.<\/p>\n\n\n\n

Phase 2 – Duration 15 Days – Contacting EC2 Abuse Team – Not much help from them<\/h3>\n\n\n\n

However, the AWS ec2 abuse team was taking a huge amount of waiting time and no useful response from the last 10 days, thus i decided to investigate myself only.<\/p>\n\n\n\n

I was not skilled enough but I stressed myself beyond my work time and got into an apache log and realized that its nothing but DDOS\/HTTP flood attack. Meanwhile, I installed many tools to block the request but found none helpful. It’s very difficult to trace HTTP flood attacks. Finally i decided to block all the IP addresses which are making requests to my server more than 1000 in a day.<\/p>\n\n\n\n

Some of the commands which helped me as below;<\/p>\n\n\n\n

Later, when I could control the DDOS attack, I tracked the IP Address using https:\/\/www.opentracker.net\/ and found that those IPs address belong to no one but Amazon itself. I was shocked. For some time, I started assuming that AWS itself is involved in this attack. Finally, I submitted my analysis report to the Ec2 abuse team. Again Abuse team took 3-4 days to locate the owner of IP addresses and it was owned by NewRelic and instead of blocking those IPaddress, they asked me to reach the NewRelic support team directly. For some time, I felt strange but I had to do it. For all these processes of the AWS Support Team, EC2 abuse team’s decision to reach out to the NewRelic support team was 15 days. Let me also, one interesting observation is – None of the team was able to help me anything rather just sending email responses and hurry to close a ticket. Here I was the myself who studied all the findings and shared with the abuse team and they asked me to reach out to the NewRelic Support team.<\/p>\n\n\n\n

Phase 3 – Duration – 20 Days – Contacting NewRelic Support team – – Not much help from them<\/h3>\n\n\n\n

However, I reached out to the NewRelic Support and NewRelic Abuse team with all the details of the IP address and their response was more shocking. Instead of finding a culprit accounts and blocking them, which you can find below;<\/strong><\/p>\n\n\n

Thanks for<\/span> reaching out - sorry for<\/span> your stressful situation. Those IP addresses do<\/span> look to be related to New Relic Synthetics monitors.\n\nYou can identify the account is which the monitors live by using the X-Abuse-Info.\n\nHere are instructions for<\/span> locating the account:\n\nhttps:\/\/docs.newrelic.com\/docs\/synthetics\/synthetic-monitoring\/administration\/identify-synthetic-monitoring-requests-your-app<\/span>\n\nPlease let<\/span> me know the account IDs that you find so that I can investigate this<\/span> further.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
I felt strange that NewRelic does not maintain the customer details of which website is monitored by which customer id. They were having domain names on which synthetic-monitoring is enabled but they were not able to trace the account. I was shocked. They sent me this email in response.<\/strong><\/p>\n\n\n
We are unable to assist you with<\/span> finding the headers or the request logs on your network. Perhaps there is a network engineer within your organization who can assist with<\/span> this<\/span>.\n\nThe following header is sent along with<\/span> every request that our Synthetics monitors make\n\nX-Abuse-Info: Request sent by a New Relic Synthetics Monitor (https:\/\/docs.newrelic.com\/docs\/synthetics\/new-relic-synthetics\/administration\/identify-synthetics-requests-your-app) - monitor id: ${MONITOR_ID_STRING} | account id: ${ACCOUNT_NUMBER}<\/span>\nWithout the above information there is unfortunately no way we can assist any further other than recommending blacklisting all of<\/span> our associated IP addresses which will block these requests.\n\nIf you have any questions about this<\/span> please let<\/span> me know.<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n\n\n\n
This blog is still under in-progress<\/p><\/blockquote>\n\n\n\n
\n
How to save HTTP requests headers, methods and body to a file?<\/a><\/blockquote>