Without giving too much detail, know that Turbine developed two proprietary script languages to help abstract the multiplayer aspects of the game and solve synchronization and messaging requirements for massively multiplayer games. Having done so, a lot of modern approaches to performance profiling fall a little flat because they come down to an inconvenient fact that the game code is written almost entirely in these two proprietary languages and the last call on the stack ends up being the call into the VM. Over the last two years, we've developed profiling and analysis tools to aid us in diagnosis and fixing problems.

That being said, knowing about a problem and coming up with workable solutions can be challenging. The era under which these games were developed is VERY different from a hardware topology standpoint. These were developed during the times of single processor servers prior to broadband and when servers were not virtualized.Turbine invented its own form of compute cloud, storage solutions, and game engine. Some architectural decisions were made to minimize memory requirements because memory was rather limited and expensive which tends toward more CPU intensive solutions. Another thing to keep in mind is that post launch the game took directions that original team could never have foreseen and therefore wrote algorithms and made architectural decisions that can be a challenge given where we are now.

An example of this is can be illustrated with feats in the game. When I asked for a specification on feats for the game during early development, the design lead at the time handed me a copy of the 3.5 edition players handbook and said "This is your specification!" Cool! That's less than 100 feats and most of those are specializations and masteries. At the time, network speeds were not as fast as they are now and serializing a player across a server boundary was considered very expensive, so we jumped through hoops to compress information on the player. We were operating under the impression that like other MMOs we would have cross server boundary combat akin to WoW and LOTRO. (This was prior to deciding upon the instanced model of content that DDO later adopted.) Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number. We did not envision a design team solving the problem of advancement at a less macro level by using feats as the workhorse they have become. Initially, those small iterative pathways were not a performance issue and no one batted an eye at that n squared algorithm. We had some coding standards to keep certain pathways from crossing to keep it from becoming a performance issue like "feats don't modify effect properties" and "effects don't modify feat properties". So that feats could cache their bonuses and not have to be re-calculated except on logon and advancement. At some point, one of the developers fixed a bug where feats and effects weren't stacking per the player's handbook crossing that stream because nobody was left from the original combat team to stop them. So then every time an effect touched one of these properties, it had to rebuild this value iterating over all the possible feats that could touch it. An n squared algorithm became n cubed. At the time, it wasn't that big of a performance hit because we didn't have 485 refreshed bonus properties and 7000+ potential feats. LOL... just gave too much detail.

So this fix is very much a fix fighting against the very architecture under which the feat system was designed. This is just one example... performance on projects this massive isn't just one big issue. I always say performance on these projects is death by a million paper cuts and only through steady applied pressure and paying our technical debts by carefully changing what is prudent and testable, we can improve performance. This change has been a long time coming with a lot of different rounds of testing and finding a lot of peculiar and stumping problems along the way. This change has been silently tried on Lamannia a few times running the new code in tandem with the old code and cross checking the results to find errors. This time, we felt like we are getting close enough to release with it and decided to turn off the old code pathways and see what issues popped up.

I'm super grateful for the fans that have kept this franchise going and we aspire to be worthy of your patience and loyalty to this product. I hope this change is successful and brings greater joy to gameplay. We are painfully aware that there is a long way to go when it comes to performance fixes on the client and server and we have to balance that against new feature development and tech rot associated with tools and technologies written 15 years ago. I hope this explanation is somewhat illuminating and I would kindly ask that those in the community not assign incompetence or malfeasance to the developers at SSG. There are many of us who were there in the early years who have come back and a lot of new faces. Everyone here really cares about the products and we really care about our players and the longevity of the franchise. We are human and make mistakes. Games like this have a burden of legacy systems and data that sometimes make it hard to anticipate the halo of changes when they are made.

I don't tend to surface often because I'm hella busy and when I worked for the games previously I spent a little too much time on the forums and got myself in trouble with work/life balance. It is nice that somebody remembered me from my "Doghouse" thread days. We still have the castle bed and four more children have enjoyed it since that time. I have really good memories of having such a collaborative working relationship with the fans of the game. I'll try to surface more often so long as we can keep the vibe constructive. However, I must say that you, my boss, and coworkers would rather me spend my time working on the game than chatting on the forums.

Some algorithms literally walk through every possible feat in the game with a certain category and check if you have that bit set which at launch was a quite small number

You're not thinking far enough back. Remember DDO's code is based on Asheron's Call, so we are talking server architecture from the mid-to-late 90s, not the early 2000s. Multi-processor servers did exist; but at the time it made more sense to use 1 cpu to run the OS/network, and the other to run the actual server application. Hence DDO was written to be single-threaded.

MSSQL, as in MicroSoft SQL? Ugh. Is that even still under support?

T-SQL is bad? I hope not. I just learned it.

Not really at DDO Time, at AC time

Oracle license have always cost an arm, a leg, an eye, a kidney, a lung and your firstborn...

The choice was simple and probably dictated by the hardware architecture used server side.

idk if ms sql has a cost to be honest

It requires licensing like any other serious Microsoft piece of software.

MS SQL licensing (on the server at least) was based on the # of CPUs the server had. IIRC, there is also licensing required for the # of users.

Funny story, I just found my original NWN disks yesterday whilst going through some old boxes. I managed to lose nearly all my old games I actually played... but not NWN.

I dont even have a working drive to put discs into.  I probably  can';t play any old games that require a disc be present anyway

I build retro PCs for fun and have a whole box full of working optical drives both SATA and PATA. Now if I could only find all my vintage games I'd be in great shape...