PG Beta - Development Roadmap (Serb)

[Karnix]

A month ago, seeing 75 people online was an amazing achievement; last night there were over 450 people online.

The company I work for (also in Florida, PS we should get some beers sometime!) has directed my group of engineers into machine learning over the past 6 months; in doing so we have several Titan V clusters setup around the office. Since they are not utilized all the time I thought I'd put PG on one of those super computers and give it a whirl. I was getting 40ish FPS (Ultra/4k/optimize mesh) in Serb, with a $3000 graphics card (currently the best gaming graphics card on the planet) and a $2000 cpu (~20000 on cpu benchmark).

Question: Just wanted to know what the current plan/prioritization was in regards to fps performance.

I love this game, and unfortunately no amount of $ can give me decent fps in town (on my machine at home which uses a 1080 TI I only get about 30 fps in Serb and it shutters all over). 30 FPS is playable, and i can get 45 if I drop the resolution; but for most people who play on low end computers, they are having issues navigating Serb (on lowest graphics setting) and I am worried about scalability.

I know this game was not intended for a million players, but if we lag at 100 people, and barely can move at 450, what will 1000 do?

Suggestion: I actually do not mind the graphics (honestly), the gameplay is amazing (and addicting :P)! I know a few months ago you mentioned that artwork/graphics would be enabled/prioritized via Steam Release funding; all I am suggesting is that the scope is tailored slightly to fix the main town that every single player has to spend a considerable amount of time in (Serb). Amazon's AWS elastic servers have come a long way, and can be integrated into Unity Game Servers, might be worth looking into. https://aws.amazon.com/gamelift/

[Mechant]

I had 2 fps during the cauldron event it felt like my computer was about to die. I am trying this https://www.reddit.com/r/projectgorgon/comments/85drp9/project_gorgon_fps_boost/

[Citan]

Optimizing Serbule is definitely a high priority, and we've got some framerate improvements ready for the next game update (which will be next week, probably). But it's a complex problem with a lot of different parts, and we'll have to iterate and improve over several months to get things settled. This next update will give a noticeable FPS boost to everyone by fixing some problems in the Serbule scene itself, but there are other issues that need to be worked on.

One issue is that our monster-movement subservers seem to become unstable when there's more than 300 people on a single map. That happened during the "poetry" player event on Sunday for the first time. Those sub-servers are designed to crash and restart cleanly, and they did so, but whenever one crashes, players in the area experience noticeable lag (and can also have weird monster behavior, such as monsters "skating" around). That lag lasts until the subserver restarts, usually less than a minute later. The Serbule sub-server crashed five or six times during the few hours around the poetry event, so that's pretty annoying.

I'm still looking into why the subserver crashed this weekend, and fixing it will probably take iterative improvements and testing. But in the short term we'll set up dual physics sub-servers for Serbule so that they can share the work and fail-over if one crashes. That configuration hasn't been tested before but we'll set it up this week and see how it behaves this weekend when populations are highest. And, of course we'll work on fixing the underlying crash problems!

We also need to optimize a lot of things, from LODs to animation-visibility ranges. There's a lot of work to do. But most of it isn't unknown work -- it's just a lot of work we need to do. So we will!

Just to be clear: it should definitely be possible to have hundreds of people on the Serbule map, but having hundreds of players ON-SCREEN is always going to be expensive. I doubt it will ever be possible to have 150+ people in town without having significant framerate drops in town. Our character-customization hooks are pretty deep (for instance, every piece of armor can be dyed separately on three layers in any of hundreds of colors), so rendering players will always be somewhat expensive. However, the "OptimizeMeshes" special optimization feature is especially useful when there's tons of players on-screen, and I encourage people to test it out! (Check the patch notes for details.) We have plans to improve that further. And as we fix other problems with CPU-bottlenecks, the pressure will ease a bit.

But part of the long-term fix is to push players to spread out more. If there's 1000 people online, they should be distributed across multiple cities, not all standing in Serbule square! To that end we've been working on making Rahu more useful as a high-level city, and we'll keep working on that in future updates, too. And we'll have to experiment with systems or content that will help player-run "events" like the poetry event succeed.

So to sum up: we'll be trying something this weekend to address the sudden FPS spikes that happen when the physics sub-server crashes. The update next week will have some FPS improvements. And the updates after that will continue to improve things. We use an iterative approach, so it's not feasible to give an exact timeline of when Serbule will be "optimized", but it will improve noticeably in steps.

[Karnix]

Thank you so much for taking time to explain your plans with us! I am really excited for these future changes and will do what I can to provide quality feedback throughout the iterations.

[Ells0430]

Optimizing Serbule is definitely a high priority, and we've got some framerate improvements ready for the next game update (which will be next week, probably). But it's a complex problem with a lot of different parts, and we'll have to iterate and improve over several months to get things settled. This next update will give a noticeable FPS boost to everyone by fixing some problems in the Serbule scene itself, but there are other issues that need to be worked on.

One issue is that our monster-movement subservers seem to become unstable when there's more than 300 people on a single map. That happened during the "poetry" player event on Sunday for the first time. Those sub-servers are designed to crash and restart cleanly, and they did so, but whenever one crashes, players in the area experience noticeable lag (and can also have weird monster behavior, such as monsters "skating" around). That lag lasts until the subserver restarts, usually less than a minute later. The Serbule sub-server crashed five or six times during the few hours around the poetry event, so that's pretty annoying.

I'm still looking into why the subserver crashed this weekend, and fixing it will probably take iterative improvements and testing. But in the short term we'll set up dual physics sub-servers for Serbule so that they can share the work and fail-over if one crashes. That configuration hasn't been tested before but we'll set it up this week and see how it behaves this weekend when populations are highest. And, of course we'll work on fixing the underlying crash problems!

We also need to optimize a lot of things, from LODs to animation-visibility ranges. There's a lot of work to do. But most of it isn't unknown work -- it's just a lot of work we need to do. So we will!

Just to be clear: it should definitely be possible to have hundreds of people on the Serbule map, but having hundreds of players ON-SCREEN is always going to be expensive. I doubt it will ever be possible to have 150+ people in town without having significant framerate drops in town. Our character-customization hooks are pretty deep (for instance, every piece of armor can be dyed separately on three layers in any of hundreds of colors), so rendering players will always be somewhat expensive. However, the "OptimizeMeshes" special optimization feature is especially useful when there's tons of players on-screen, and I encourage people to test it out! (Check the patch notes for details.) We have plans to improve that further. And as we fix other problems with CPU-bottlenecks, the pressure will ease a bit.

But part of the long-term fix is to push players to spread out more. If there's 1000 people online, they should be distributed across multiple cities, not all standing in Serbule square! To that end we've been working on making Rahu more useful as a high-level city, and we'll keep working on that in future updates, too. And we'll have to experiment with systems or content that will help player-run "events" like the poetry event succeed.

So to sum up: we'll be trying something this weekend to address the sudden FPS spikes that happen when the physics sub-server crashes. The update next week will have some FPS improvements. And the updates after that will continue to improve things. We use an iterative approach, so it's not feasible to give an exact timeline of when Serbule will be "optimized", but it will improve noticeably in steps.

I was at the poetry reading on Sunday and had my camera on the event the whole time my framerate stayed pretty much in the 20s and 30s with a few dips into the red but they were so fast I couldn't get a number. I was surprised it handled it so well considering my computer is not a huge beast or anything(i5, 8gb ram, geforce gtx 750 ti) I am using "OptimizeMeshes" I hope this info is helpful. I will check it out again this weekend to see if it is stays the same. It was a fun time that's for sure!

Thanks for all your work I imagine you all haven't had much sleep or free time these past couple weeks. I hope you get a nice relaxing night off soon :)

[Mechant]

If rendering a player costs that much, why don't you add something like disable other character visualization as an option, and why there isn't a character view distance option, these may already improve fps a lot.

[Citan]

Players are already hidden if they're too far away. (And there's also a distance where they're drawn, but not animated, as a separate optimization.)

As for why there's no "disable customizations" option, those customizations are built deeply into our code (and heavily optimized), they're not something that's just tacked on. It would be a huge amount of work for something that looks pretty crappy. We'd need a completely separate set of shaders and mesh-bakers and so on, and the result would be that all players would have gray-tone armor on, since the base armor colors are all gray. And players' skin tones would be lost, faces and hairstyles wrong, etc. It doesn't really make sense here.

I guess the important thing to point out here is that we've already done a lot of basic optimizations! I know it doesn't seem like it's optimized, but we've done most of the obvious things you can think of. The thing is, MMOs are not simple games. Optimizing one takes a lot of work.

[Madmatx]

Players are already hidden if they're too far away. (And there's also a distance where they're drawn, but not animated, as a separate optimization.)


Is that why we see the player "scarecrows" skating around sometimes in the distance?

[Crissa]

Is that why we see the player "scarecrows" skating around sometimes in the distance?

Yes. The default poses are kinda funny. Some are lying down ^-^

[Karamasha]

Players are already hidden if they're too far away. (And there's also a distance where they're drawn, but not animated, as a separate optimization.)

As for why there's no "disable customizations" option, those customizations are built deeply into our code (and heavily optimized), they're not something that's just tacked on. It would be a huge amount of work for something that looks pretty crappy. We'd need a completely separate set of shaders and mesh-bakers and so on, and the result would be that all players would have gray-tone armor on, since the base armor colors are all gray. And players' skin tones would be lost, faces and hairstyles wrong, etc. It doesn't really make sense here.

I guess the important thing to point out here is that we've already done a lot of basic optimizations! I know it doesn't seem like it's optimized, but we've done most of the obvious things you can think of. The thing is, MMOs are not simple games. Optimizing one takes a lot of work.


Besides Serbule keep my game runs really good, im looking forward to further optimizations.