Seeking assistance with increased save times (paid)

Archaaz

Well-Known Member
Sep 17, 2016
128
16
My RunUO 2.2 shard has been experiencing issues with increased save times since about February. I have been running the shard on a VPS since early 2017. The save times were 0.4 to 0.7 seconds (usually around 0.5). Earlier this year, after adding several new scripts over the course of a few months, I noticed that save times had increased by about a second, typically they were about 1.2 to 1.5, and lately have increased to about 1.5 to 2.0, sometimes more. Occasionally, a save will take 8-10 seconds. I have been monitoring the item and mobile counts, and they are holding steady at about 131,000 items and 14,000 mobiles, give or take. Unfortunately, I did not notice the precise moment when this began to happen. It seems to have been somewhat gradual.

I am running the shard on a VPS with 2 cores and 4 GB RAM. Thinking it could be an issue with the VPS, I increased this to 4 cores and 8 GB RAM, but have the same problem.

Today, I uploaded a backup of the shard I had made prior to experiencing these issues. The save times on that server are 0.4 to 0.7 (usually 0.5). The item and mob count are slightly less- 125,938 items and 13971 mobiles- but not enough to explain the increase.

This has lead me to conclude that the increased save times are a result of something I added (or removed) or some change I made to a script, or otherwise messed up in the last few months.

I am hoping to fix the issue without having to revert to the older version and try to readd and respawn everything I have added in the last five or six months (I did a lot of deco and added quite a few spawners). I am not sure there is a way to trace such a problem, but I would greatly appreciate any help. I am willing to pay (and will do so promptly), though we will need to discuss the price. Thanks in advance.
 

bojangles2872

Well-Known Member
Jul 29, 2018
74
8
Start with your Saves folder. Compare it to your older one and see if there is something in there that is not in your older or something that is way bigger. This would at least help you narrow down what is causing the long save times.
 
  • Like
Reactions: Archaaz

Archaaz

Well-Known Member
Sep 17, 2016
128
16
Start with your Saves folder. Compare it to your older one and see if there is something in there that is not in your older or something that is way bigger. This would at least help you narrow down what is causing the long save times.
Thank you very much! I had actually decided earlier today to replace the save folder in a test server based on my current server with the save folder from the back up server. This fixed the problem, and returned saves back to my the 0.4 to 0.7 range. This is all well and good, but may not have addressed the underlying issue. It may still creep back up to the increased save levels due to an underlying issue.

On your advice I just check the size of each folder in the save folder in the current folder. All were the same size, or near to it, except for Mobiles. The Mobiles folder in the back up server save folder was at 135 MG, and the Mobiles folder in my current server was 445 MG. This is interesting because the amount of mobiles on both servers is in the ballpark of 14,000. I have not experienced an influx of new players, and currently only have two somewhat active players, and myself. In the past I had maybe 20-30 players, though I think that most have not played for over a year. I have made some changes to mobs, which might account for the save times, though I am not sure how just yet. This is good news insomuch as deco and other items is concerned.

Again, thank you for your advice!
 

zerodowned

Moderator
ServUO Developer
Jun 28, 2014
1,775
222
I had similar issues when running my own shard.
What are the delays between saves?

Not sure about the file path for Runuo anymore but for Servuo it's:

ServUO\Scripts\Misc\AutoSave.cs

or

ServUO\Config\AutoSave.cfg

...depending on what version of Servuo you're using
 
  • Like
Reactions: Archaaz

Archaaz

Well-Known Member
Sep 17, 2016
128
16
I had similar issues when running my own shard.
What are the delays between saves?
Thanks! It is the same for RunUO. The delay is set to 5 minutes. The warning was set to 15 seconds, but that line is commented out, and has been changed to zero.

I created a test server based on the actual server. I deleted all BaseCreatures on all but the internal server and the size of the Mobiles folder was reduced from 445 MB to about 6.5 MB, and save times were back to 0.3 to 0.6. I am considering this "fix" for the main server as well; deleting or respawing all BaseCreatures except pets and a few special classes of mobiles. My thoughts are that I can do this periodically when needed, without affecting PlayerMobiles.

The test server Mobile save folder gains about 1 MB every two days or so. The current server was doing likewise but has increased significantly in the last couple of weeks.

What did you do to fix the problem?
 

bojangles2872

Well-Known Member
Jul 29, 2018
74
8
I think there is something in xmlspawner to where you can make the mobiles spawn only when a player is within a certain range. Would be worth looking into and testing it if it exists.
 
  • Like
Reactions: Archaaz

Archaaz

Well-Known Member
Sep 17, 2016
128
16
I think there is something in xmlspawner to where you can make the mobiles spawn only when a player is within a certain range. Would be worth looking into and testing it if it exists.
Thanks! I will definitely look into that.
 

DaeronModaen

Member
Apr 14, 2018
17
4
40
I think there is something in xmlspawner to where you can make the mobiles spawn only when a player is within a certain range. Would be worth looking into and testing it if it exists.
that would be SmartSpawning property. you don't want to use smart spawning if you are using sequential spawns or player triggers on the xml spawner. smart spawning seems to mess with those things. for your standard spawn it is okay. Im not really sure that will help with the save times, but who knows, it might. it would be worth a try.

Unfortunately you would probably have to examine what you have recently added to really get down to the bottom of what the cause of the increased server save time. Things like server utilities, and monsters that spawn other monsters could contribute to it. It would also probably be helpful to view the server console as it does a save to see if it is having trouble creating a report or doing something else. if for whatever reason it cannot create one of the saves, that would cause the save to lag some increasing the save time.
 
  • Like
Reactions: Archaaz

bojangles2872

Well-Known Member
Jul 29, 2018
74
8
I dont know much about the smartspawning but I remembered seeing it and figured it would be worth mentioning. It does appear to be something with his monsters though, maybe one way to narrow it down could be to spawn one facet at a time and see if those save times increase dramatically when one of them is spawned. This would help narrow down what to look for, also only start with the normal spawns and nothing custom.
 
  • Like
Reactions: Archaaz

Archaaz

Well-Known Member
Sep 17, 2016
128
16
that would be SmartSpawning property. you don't want to use smart spawning if you are using sequential spawns or player triggers on the xml spawner. smart spawning seems to mess with those things. for your standard spawn it is okay. Im not really sure that will help with the save times, but who knows, it might. it would be worth a try.

Unfortunately you would probably have to examine what you have recently added to really get down to the bottom of what the cause of the increased server save time. Things like server utilities, and monsters that spawn other monsters could contribute to it. It would also probably be helpful to view the server console as it does a save to see if it is having trouble creating a report or doing something else. if for whatever reason it cannot create one of the saves, that would cause the save to lag some increasing the save time.
I am using mostly Premium Spawners rather than XML Spawners. Would Smart Spawning work with those?

Thanks for the advice! I released a new client back in November with some new/modified maps and artwork. At that time I made a back up of the server. I loaded this a few days ago and let it run on the VPS. The Mobile save folder on that server was 135 MB. It has increased to 137 MB over the last few days, leading me to suspect that the issue has been going on for some time. I am going to monitor it for a few more days.

I am going over scripts and changes made since November, though I am not sure quite how far back to go. I am considering reverting to this older back up server, adding the Items save folder from my current server, deleting or re-spawning BaseCreatures (except any pets on maps other than the internal, if possible), and then vetting the newer scripts as I re-add them. I am not sure this will necessarily fix the underlying issue, but it might be a better starting place.

I compared the saves in the console on my current server with my back up server, and they seem to be functioning correctly and in the same manner. I cannot detect any abnormalities, but will continue to monitor it.
Post automatically merged:

I dont know much about the smartspawning but I remembered seeing it and figured it would be worth mentioning. It does appear to be something with his monsters though, maybe one way to narrow it down could be to spawn one facet at a time and see if those save times increase dramatically when one of them is spawned. This would help narrow down what to look for, also only start with the normal spawns and nothing custom.
Thanks! I will try respawing one facet at a time. I am currently only using two facets, Felucca and Trammal. I am using a custom map for Trammal, and only a portion of it, so not many mobs on it. The bulk are on Felucca. I am not quite sure how to spawn normal spawns vs custom.
 

bojangles2872

Well-Known Member
Jul 29, 2018
74
8
Custom spawns you would have to add in yourself. If you havent done anything like that and still keep getting these long save times there may be something else wrong thats causing the problem other than your actual spawns.
 
  • Like
Reactions: Archaaz

Archaaz

Well-Known Member
Sep 17, 2016
128
16
Update: I found the issue. Taking the advice offered here, I began deleting mobs by facet, and then by type. It was with a certain type of custom mob. After deleting these mobs the file was reduced to a little over 7 MB, and save times are at 0.26 to 0.4 seconds. Thanks everyone for your help with this! I really can't thank you all enough.
 
Last edited:

DaeronModaen

Member
Apr 14, 2018
17
4
40
what kind of mob was it? it might be worth while to share a bit of info about it, in the case someone else has a similar issue and needs help fixing it. glad you got it figure out.
 
  • Like
Reactions: Archaaz

Archaaz

Well-Known Member
Sep 17, 2016
128
16
what kind of mob was it? it might be worth while to share a bit of info about it, in the case someone else has a similar issue and needs help fixing it. glad you got it figure out.
It was a custom mob based off of TalkingBaseCreature. Still not sure what exactly the source is. I am working with someone else to isolate the precise cause of the issue. I will post more when we discover the cause. Thanks again for the help!