Meursault wrote on Apr 9^th, 2016 at 2:02pm:

Looked like Qwest to me also when I ran a few traces right after the move.  The last hop before I hit their server was a Qwest address.
I've lost the host IP they provided and I'm not really interested in looking it up, so I can't show you my traceroute.  You might be using the old host address?  It does not look familiar to me, but then memory might not be serving.

Meursault wrote on Apr 9^th, 2016 at 2:02pm:


now that's interesting... Do you trace on gls.ddo.com or on an IP ?

C:\Users\Guillaume>tracert gls.ddo.com

Détermination de l'itinéraire vers gls.ddo.com [198.252.160.23]
avec un maximum de 30 sauts :

  1    <1 ms    <1 ms    <1 ms  <private ip>
  2    16 ms    15 ms    16 ms  <access router ip>
  3    16 ms    16 ms    15 ms  10.123.109.138
  4    17 ms    17 ms    16 ms  ae41-0.niaub201.Aubervilliers.francetelecom.net [193.252.98.161]
  5    19 ms    23 ms    23 ms  81.253.184.182
  6    19 ms    19 ms    19 ms  tengige0-14-0-9.auvtr1.Aubervilliers.opentransit.net [193.251.242.217]
  7    98 ms    98 ms    99 ms  tengige0-6-1-0.nyktr1.NewYork.opentransit.net [193.251.240.202]
  8    95 ms    95 ms    94 ms  nyc2-brdr-01.inet.qwest.net [63.146.27.5]
  9    96 ms    96 ms    96 ms  ewr-cntr-11.inet.qwest.net [205.171.17.2]
 10   105 ms   105 ms   106 ms  206.103.215.50
 11    93 ms    93 ms    92 ms  63.236.3.130
 12     *        *        *     Délai d'attente de la demande dépassé.
 13   105 ms   105 ms   106 ms  198.252.160.23
 14   106 ms   106 ms   105 ms  198.252.160.23

Itinéraire déterminé.

C:\Users\Guillaume> 

ARIN sais that 63.236.3.130 belongs to Qwest.
hop 12 is the (in)famous Turbine border router that spend most of it's time not answering to traceroutes and when answering gives an ip in the 10.0.0.0/4 private range.

There is no reasons why I would move from France Telecom backbone ( OpenTransit ) to Qwest if the Datacenter belongs to PNAP. If it was the case I'd move to PNAP backbone at hop 8.

if you go by PNAP up to the (in)famous Turbine border router it means that they have done the two circuit trick for DDO too ( which is logical... it wouldn't be tied to a game, but to everything in that datacenter ), and that each circuit is using a different carrier ( one PNAP, the other Qwest ).

I actually visit wayfinder every 2-3 days and it seems its getting fucking lag everywhere, every single minute.

I guess my first guess was not far off with "datacenter move" being a simple server swapping.

I'm gooooooood!

Flav wrote on Apr 11^th, 2016 at 4:24am:


It isn't really necessary to "kill virtualization".  There is nothing intrinsically better about a hardware solution over a virtual solution.  A poorly designed hardware solution will suck ass.  As will a poorly designed virtual solution.  A well designed hardware solution can be amazing.  As can a well designed virtual solution.  The only difference is that 95% of systems architects understand proper hardware solution design wheraas maybe 15-20% have a handle on proper virtualization design.  The average number of years of experience with hardware solutions is probably 3-4 times more than virtuals.  This lack of available experience often flows down through the whole project team.  Often the idea of preceding a project of this sort with training and education at a certified training center is skipped.  So the risk associated with virtual solutions is higher. 

We moved our apps to virtuals and they have never performed better.  We get almost twice the density we were getting on hardware.  But our hardware was ancient (purchased in 2007/2008).  And we did 6 months of testing the apps starting with functional tests through full stress tests to determine the proper configuration before we did the transition.  And we did a soft cut over where we migrated the customer base over the course of 90 days vs. a hard cut like Turbine did.  So we could adjust as actual user load started increasing.  Just in case our testing missed anything.  (Real users do shit you can't predict 100%.  No matter how good your test scripts are.)

This isn't necessarily a decision error in hardware vs virtual.  I believe it to be more of an execution error.

Regardless, they need to do something.  The lag is still killing raids.  I don't have much problem in 6 man content, but geez.  The raids are annoying as hell.

Asheras wrote on Apr 11^th, 2016 at 9:21am:


I will agree with you... in general terms.

In DDO Case it was just stupid to move towards virtualization. The software was is designed for that, and the architecture is probably not designed either.
It's probably would have been cheaper and less damaging to just replace the hardware and be done with it during the move instead of going to virtualized stuff.

noamineo wrote on Apr 11^th, 2016 at 3:15pm:

We'd wonder that if we didn't know Turbine, sure. Their corporate structure essentially guarantees that they didn't know they were going virtual until the bean counters decided. At which time, it was far too late for testing.

Flav wrote on Apr 12^th, 2016 at 3:22am:

In this case it might have actually done what it was attempting to do.  The player load does not seem to be any part of the issue.

Frank wrote on Apr 12^th, 2016 at 7:07am:


I should have said it another way...
The process that crashed was the communication process between the server ( alarm gathering server ) and the telecom equipment ( that generated the alarms )
with the server connected to a single test telecom equipment, that generated test alarms when needed by the test plan, everything was fine.
with the server connected to 6 telecom equipment, that generate live alarms from a real telecom network, the process crashed in a matter of seconds.
with the server connected back to the single test telecom equipment, and a frustrated flav in control of all the parts, the process crashed in a matter of minutes.

It took one week for the developer that got assigned to me to fix the problem...
at the rate of one flav generated crash in the morning, one new version of the daemon that crashed late in the afternoon.
Once the daemon survived a telecom equipment restart ( and the 100 pages of alarms it generates in such an event )
as well as a flav tinkering with other applications and settings at the same time, it was considered fit for duty.
( that's the best I could come up with a test bed sized single telecom exchange to simulate a real 6 exchange network )

player load is similar... you try the thing with 30 players and everything is fine... you try it with 300 and everything go to hell.