Fun day at work..

[Rhyker]

Good day,

/startrant

Today is Saturday, my day off from work, I though I'd relax for the day, and just chill out, boy was I wrong. I received a call at about 8:30AM saying that the network at work had gone down, I though it was a simple issue, because something like that had happened before (Janitor knocked the power cords out of one of our network gateways) so I was like, "PERSONSNAME, go downstairs and check if the gateway is unplugged again," So he goes down and it's still plugged in, so I decide to try and RDP into one of the servers in our winblows rack (I honestly don't know why we need them, we could achieve the same thing, for half the cost, with linux...but I digress).

That's when all the fun starts. My connection keeps timing out, so I figure it's on my end because the racks have a dedicated 10gbit line each. So I check everything here at home and all is well, so I figure it's something that has gone wrong with the gateway, so I get up, get my ass to work and start checking to make sure all the lines are connnected...etc.. . I then go to the on-rack monitor to see if something happened and caused it to shut down, restart...blah blah. Server is up and running fine, no processes are running that shouldn't be, so I take a look at the network load....shows that we are using 80-100% of our downstream data, which tells me that something is seriously wrong. ((Our linux racks that I maintain are better equiped, I'll explain later)) At work here, we set up a load balancing arrangement, because the employees love to download random shit, stream movies, etc... So if our downstream gets choked at 75% it will switch to another server to ease the pressure on our gateway. For it to have been running at 80-100% for the last 3 hours, means that ALL servers are choked up. So my first order of business is to NULL the IP's of the choked servers, and assign new ones. So that went fine, and then I had to do some digging to find out what in the hell was happening. I go in and check the intranet access logs, and all is fine, so I check outside connections and notice a little bit of an issue.((I've got our logs set up to log the IP's of any outside connection sending data packets larger that X bytes per second)) Low and behold

click spoiler to show IP's

Spoiler

103.230.84.239
103.24.3.198
103.241.0.100
103.4.52.150
103.7.59.135
104.28.1.39
107.150.58.84
107.181.174.84
108.166.181.239
108.175.149.16
108.61.123.204
108.61.208.120
108.61.210.194
108.61.252.184
108.61.63.78
109.120.150.246
109.120.183.106
109.127.8.242
109.229.210.250
109.229.36.65
109.235.59.44
113.108.204.30
116.193.77.118
118.145.2.189
119.59.120.16
120.63.157.195
122.155.7.232
123.30.129.179
128.199.154.17
128.199.230.64
128.210.157.251
131.72.136.241
142.0.47.86
144.76.240.102
144.76.240.104
144.76.240.109
144.76.240.110
146.185.174.81
146.185.253.129
146.185.253.136
148.251.216.83
148.251.248.9
151.248.123.110
151.97.190.239
159.253.43.120
162.144.103.191
162.213.250.240
173.230.253.193
173.242.112.135
173.243.112.148
173.243.112.187
173.243.112.235
173.249.152.23
176.9.236.6
177.140.22.150
177.55.106.140
178.18.17.205
178.217.187.126
178.63.62.55
178.75.246.57
180.182.234.200
181.41.210.61
181.47.38.206
184.168.47.225
185.11.145.174
185.12.44.5
185.16.212.70
185.24.233.117
185.24.234.108
185.25.117.49
185.31.160.83
185.38.84.18
185.8.105.60
185.8.105.61
186.202.177.149
186.64.120.104
187.174.252.247
188.219.154.228
188.247.135.53
188.247.135.58
188.247.135.74
188.247.135.99
190.128.29.1
190.14.37.27
190.15.192.25
190.97.165.224
192.185.216.221
192.254.79.106
192.64.11.244
192.64.9.116
192.95.12.34
193.106.177.243
193.107.17.56
193.107.19.24
193.109.68.87
193.120.55.242
194.15.112.29
194.15.112.30
197.242.79.34
198.15.117.228
198.15.117.240
198.15.127.170
198.245.202.92
198.27.77.8
198.50.180.167
198.50.198.162
198.50.232.214
198.58.83.57
199.175.48.194
199.187.129.193
199.201.121.185
199.231.186.170
199.7.234.100
200.111.67.91
202.142.215.16
202.29.230.198
203.123.60.144
203.170.193.23
204.16.169.2
204.95.99.66
209.200.232.14
209.217.235.17
209.99.40.224
210.211.108.215
212.225.213.253
212.44.64.202
212.59.240.74
213.136.79.120
213.147.67.20
216.176.100.240
216.215.112.149
217.110.7.181
217.23.138.114
222.29.197.232
23.227.196.23
23.227.199.91
23.227.199.93
23.235.201.32
23.252.120.143
23.253.71.170
31.148.219.85
31.170.107.182
31.210.118.212
31.220.2.150
31.24.30.76
31.24.30.81
31.7.63.146
37.0.123.109
37.0.123.133
37.0.123.149
37.0.123.161
37.0.123.191
37.0.123.204
37.0.123.226
37.0.123.84
37.0.124.118
37.0.124.145
37.0.124.35
37.0.127.107
37.0.127.108
37.0.127.110
37.0.127.112
37.0.127.115
37.0.127.96
37.140.195.148
37.143.11.189
37.58.49.143
37.59.217.219
37.59.68.14
41.186.24.58
41.71.188.2
46.102.246.202
46.149.111.39
46.166.131.154
46.166.145.113
46.183.220.75
46.183.220.76
46.183.220.94
46.183.221.47
46.183.221.54
46.183.221.63
46.183.221.74
46.19.143.249
46.21.157.219
46.22.211.47
46.4.150.111
5.101.153.74
5.199.171.20
5.199.171.32
5.63.158.126
54.191.43.114
58.195.1.4
59.157.4.2
60.13.186.5
62.173.151.82
63.143.50.250
64.127.71.73
64.32.12.8
64.85.233.8
65.200.132.20
65.75.164.128
66.49.132.125
69.175.105.170
69.194.235.103
72.52.4.121
77.40.30.111
78.110.175.78
78.110.9.77
78.138.104.167
78.153.148.113
78.157.207.44
80.82.64.200
81.4.108.109
81.88.57.96
82.131.180.72
83.15.254.242
83.69.233.121
87.107.121.131
87.236.210.110
87.236.210.124
87.236.215.103
87.246.143.242
87.254.167.37
89.248.166.144
89.36.31.215
89.46.251.146
89.46.251.158
89.46.251.169
91.197.129.190
91.213.8.191
91.218.121.136
91.226.212.170
91.227.18.17
91.228.160.170
91.228.160.201
91.230.60.107
91.230.60.232
91.239.15.219
92.53.119.248
92.53.124.62
93.171.205.11
93.171.205.39
93.174.88.3
93.186.120.112
93.186.120.42
94.102.50.48
94.102.53.158
94.102.53.160
94.102.60.178
94.102.63.137
94.102.63.143
94.102.63.153
94.103.36.55
95.181.178.177
95.211.241.171
96.30.27.132
98.131.185.136
99.181.5.83

All these IP's are sending massive amounts of data to the network. FAR FAR FAR more than is normally sent by anything else. So I decided to see just how much they were sending, so I re-assigned the gateway IP to one of my personal Linux dedi's that is running a 10 gbit dedicated line, and dropped all my IPtables rules to allow all traffic, and this is what I came up with after running a dstat for a minute or so

Spoiler

[7l----system---- ----total-cpu-usage---- ------memory-usage----- -net/total-
date/time |usr sys idl wai hiq siq| used buff cach free| recv send
20-09 08:59:33| 0 0 100 0 0 0|27.6M 0 144M 852M| 0 0
20-09 08:59:34| 0 0 100 0 0 0|27.6M 0 144M 852M| 117M 197B
20-09 08:59:35| 0 0 100 0 0 0|27.6M 0 144M 852M| 136M 446M
20-09 08:59:36| 0 0 100 0 0 0|27.6M 0 144M 852M| 144M 120M
20-09 08:59:37| 2 0 98 0 0 0|32.1M 0 144M 848M| 131M 620M
20-09 08:59:38| 3 1 96 0 0 0|35.6M 0 144M 844M| 743M 12M
20-09 08:59:39| 1 1 98 0 0 0|36.4M 0 144M 844M| 177M 49M
20-09 08:59:40| 2 4 94 0 0 0|36.3M 0 144M 844M| 31M 288M
20-09 08:59:41| 4 7 89 0 0 0|36.6M 0 144M 843M| 64M 577M
20-09 08:59:42| 6 14 81 0 0 0|36.6M 0 144M 843M| 104M 204M
20-09 08:59:43| 5 15 81 0 0 0|36.1M 0 144M 844M| 105M 499M
20-09 08:59:44| 3 5 92 0 0 0|36.4M 0 144M 844M| 37M 316M
20-09 08:59:45| 3 6 91 0 0 0|36.4M 0 144M 844M| 55M 728M
20-09 08:59:46| 4 4 92 0 0 0|41.0M 0 144M 839M| 489M 16M
20-09 08:59:47| 3 1 97 0 0 0|32.2M 0 144M 848M| 125M 289M
20-09 08:59:48| 3 0 97 0 0 0|35.9M 0 144M 844M| 655M 11M
20-09 08:59:49| 3 5 92 0 0 0|36.6M 0 144M 843M| 37M 331M
20-09 08:59:50| 4 10 85 0 0 0|36.4M 0 144M 844M| 78M 740M
20-09 08:59:51| 5 12 82 0 0 0|36.7M 0 144M 843M| 100M 851M
20-09 08:59:52| 6 15 80 0 0 0|36.5M 0 144M 843M| 191M 226M
20-09 08:59:53| 5 13 82 0 0 0|40.2M 0 144M 840M| 78M 638M
20-09 08:59:54| 5 3 92 0 0 0|32.1M 0 144M 848M| 180M 15M
20-09 08:59:55| 3 1 97 0 0 0|35.3M 0 144M 845M| 749M 13M
20-09 08:59:56| 1 0 99 0 0 0|35.9M 0 144M 844M| 719M 448M
20-09 08:59:57| 4 7 88 0 0 0|36.6M 0 144M 843M| 66M 989M
20-09 08:59:58| 5 11 84 0 0 0|36.1M 0 144M 844M| 106M 353M
20-09 08:59:59| 5 10 85 0 0 0|36.7M 0 144M 843M| 99M 771M
20-09 09:00:00| 5 10 85 0 0 0|36.5M 0 144M 843M| 111M 928M
20-09 09:00:01| 4 5 91 0 0 0|42.3M 0 144M 838M| 24M 12M
20-09 09:00:02| 3 1 92 3 0 0|31.4M 0 143M 849M| 89M 573M
20-09 09:00:03| 1 0 99 0 0 0|34.1M 0 143M 846M| 738M 10M
20-09 09:00:04| 3 0 96 1 0 0|35.7M 0 143M 845M| 822M 271M
20-09 09:00:05| 2 3 95 0 0 0|35.4M 0 143M 845M| 22M 182M
20-09 09:00:06| 3 6 92 0 0 0|36.5M 0 144M 844M| 42M 435M
20-09 09:00:07| 3 6 91 0 0 0|36.5M 0 144M 844M| 55M 493M
20-09 09:00:08| 4 8 89 0 0 0|36.6M 0 144M 844M| 63M 420M
20-09 09:00:09| 4 12 84 0 0 0|36.3M 0 144M 844M| 106M 872M
20-09 09:00:10| 4 6 90 0 0 0|36.4M 0 144M 844M| 62M 426M
20-09 09:00:11| 5 6 89 0 0 0|41.9M 0 144M 839M| 54M 385M
20-09 09:00:12| 4 2 93 1 0 0|31.5M 0 144M 849M| 160M 12M
20-09 09:00:13| 3 1 96 0 0 0|35.7M 0 144M 845M| 742M 13M
20-09 09:00:14| 2 3 96 0 0 0|36.5M 0 144M 844M| 23M 276M
20-09 09:00:15| 4 10 86 0 0 0|36.5M 0 144M 844M| 86M 592M
20-09 09:00:16| 4 9 87 0 0 0|36.3M 0 144M 844M| 67M 390M
20-09 09:00:17| 5 10 86 0 0 0|36.5M 0 144M 844M| 82M 801M
20-09 09:00:18| 3 6 90 0 0 0|36.5M 0 144M 844M| 56M 859M
20-09 09:00:19| 2 3 95 0 0 0|36.2M 0 144M 844M| 21M 279M
20-09 09:00:20| 3 7 89 0 0 0|36.4M 0 144M 844M| 64M 655M
20-09 09:00:21| 5 4 91 0 0 0|38.4M 0 144M 842M| 765M 19M
20-09 09:00:22| 3 0 97 0 0 0|35.4M 0 144M 845M| 749M 12M

Which is halarious, as they were using my own Dedi against me. It was partly my fault, as I host an open DNS resolver on there, and forgot to disable a couple things, thus DNSAMP was being used to attack my work in addition to what looks like ZEUS bots(Not done Nmap'ng all the hosts) -.- .

Anyways, I just got home about 20 minutes ago, I'm tired and slightly pissed off that I was the only person called in to deal with this. We have an IT DEPARTMENT!! for a reason. Anyways, thought I'd post about my super happy fun day.

/endrant

Cheers,

Rhyker

 

[MEX_cuz_I_CAN]

Have a cold one! If you don't drink alcohol have some iced tea or whatever you like. Looks like you learned a lesson hah. I'm having an iced protein shake after 13-mile bicycle ride but later tonight or tomorrow I'll have adult beverages.