Help for Adventure's mysterious corpse lag (runuo)

Finaltwist

Finaltwist

'Greetings all,

we've had a mysterious lag issue in ultima adventures for some time now - whenever a player dies, the entire server will lag for some time. after some experiementing around the same was found to happen when a player looted his corpse. we dug in more and realized it was directly related to the number of items the player had when he died/when he loots. the more items in the pack, the longer the lag.

We did some looking around but can't find any cause for this in corpse.cs both in core of in server scripts. something is making the server lag whenever items are transfered to/from corpses. HAs anyone encountered this? you can get the entire server code by downloading the resource....

If anyone can help fix this, ill be incredibly grateful.. hell ill even add a statue of that person in the game, commemorating the heroic deeds of the person!!!
any help is appreciated.
 
Couple of random questions:

Is it when looting manually or with a script of any kind?

If a staffer loots the corpse does it cause lag then?
 
for anyone who doesnt want to open the package, here are relevant sections of code
public virtual void Open( Mobile from, bool checkSelfLoot )
{
if ( from.AccessLevel > AccessLevel.Player || from.InRange( this.GetWorldLocation(), 2 ) )
{
#region Self Looting
bool selfLoot = ( checkSelfLoot && ( from == m_Owner ) );
if ( selfLoot )
{
if ( from.QuestArrow != null ){ from.QuestArrow.Stop(); }
List<Item> items = new List<Item>( this.Items );
bool gathered = false;
bool didntFit = false;
Container pack = from.Backpack;
DeathRobe robe = from.FindItemOnLayer( Layer.OuterTorso ) as DeathRobe;
if ( robe != null )
{
robe.Delete();
}
for ( int i = 0; !didntFit && i < items.Count; ++i )
{
Item item = items;
Point3D loc = item.Location;
if ( (item.Layer == Layer.Hair || item.Layer == Layer.FacialHair) || !item.Movable || !GetRestoreInfo( item, ref loc ) )
continue;
if ( m_EquipItems.Contains( item ) && from.EquipItem( item ) )
{
gathered = true;
}
else if ( pack != null && pack.CheckHold( from, item, false, true ) )
{
pack.AddItem( item );
item.Location = loc;
gathered = true;
}
else
{
didntFit = true;
}
}
SetFlag( CorpseFlag.Carved, true );

Item tomb = new TombStone( m_Owner, m_Killer);
tomb.MoveToWorld( from.Location, from.Map );
ProcessDelta();
SendRemovePacket();
ItemID = Utility.Random( 0xECA, 9 ); // bone graphic
GumpID = 0x2A73;
DropSound = 0x48;
Hue = 0;
Weight = 5; // final, so we can bury our bodies
ProcessDelta();
if ( gathered && !didntFit )
{
from.PlaySound( 0x3E3 );
from.SendLocalizedMessage( 1062471 ); // You quickly gather all of your belongings.

return;
}
if ( gathered && didntFit )
from.SendLocalizedMessage( 1062472 ); // You gather some of your belongings. The rest remain on the corpse.
}
#endregion
Click to expand...


In the above for example, if we simply comment out pack.AddItem(item); (no items are moved to the players pack during autoloot) the lag doesn't happen

The lag appears to happen upon moving the item out of and into the corpse. A player did a few tests - using a script that quickly moves 200 1gp coins to a corpse and out of a corpse and there was lag.

Falkor: Gm's don't lose any items to body on death, so autoloot isn't applicable to them as no items every move in/out of their corpse. If a staffer opens a corpse that hasn't been looted yet, you can see all the items, and there is no lag.
Post automatically merged:

here are both the scripts/corpse.cs and server (core) container.cs
 

Attachments

  • Corpse.cs
    40.1 KB · Views: 11
  • Container.cs
    41 KB · Views: 11
I wonder if the lag isn't corpse-related then but somewhere in the player backpack script? By that point in the script all the checks have been done so they can't be the source of the lag; simply moving the resulting item causes lag and the next thing downstream is the player's pack script.
 
but same applies on death... when all the items are moved from the pack to the corpse. that causes lag...
Post automatically merged:

the entire server code is in the resource, like i said, this stumps me
 
Have you tried commenting out the whole tombstone block? Maybe try to isolate where in the script the problem presents itself. I'd probably run the code in a test environment and add a breakpoint on that Open method and step through it.

Also shouldn't your loop be

for ( int i = 0; !didntFit && i < items.Count-1; ++i )

The list you are enumerating is zero based so if you run to Items.count, you exceed the list bounds?
 
Tempus said:
Have you tried commenting out the whole tombstone block? Maybe try to isolate where in the script the problem presents itself. I'd probably run the code in a test environment and add a breakpoint on that Open method and step through it.

Also shouldn't your loop be

for ( int i = 0; !didntFit && i < items.Count-1; ++i )

The list you are enumerating is zero based so if you run to Items.count, you exceed the list bounds?
Click to expand...
Yes, tombstone was removed and was not the cause of the lag.

I'll try the loop and report back... thanks for suggestion!
Post automatically merged:

regarding
for ( int i = 0; !didntFit && i < items.Count-1; ++i )

if there is only one item in the items list, wouldn't this change make it so the single item is skipped? If there is one element in the list (one item in the pack), then count == 1. using your sugested edit above would essentially just skip that one and final item?
 
Last edited:
Finaltwist said:
Yes, tombstone was removed and was not the cause of the lag.

I'll try the loop and report back... thanks for suggestion!
Post automatically merged:

regarding
C#: 
for ( int i = 0; i < items.Count-1; ++i )

if there is only one item in the items list, wouldn't this change make it so the single item is skipped? If there is one element in the list (one item in the pack), then count == 1. using your sugested edit above would essentially just skip that one and final item?
Click to expand...

Actually, I am wrong in this case. I hadn't noticed the use of the pre incrementor for i instead of a post incrementor in the loop.


So
  • If you remove the entire SelfLoot section the lag isnt there.
  • The SelfLoot code block contains other code beyond what posted. Did you eliminate that code?

I think you need to start looking at what else you have changed in the Open() method from the original Odyssey script. For example you have moved the block

C#: 
if ( gathered && !didntFit )

to be checked after
C#: 
ProcessDelta();
SendRemovePacket();
ItemID = Utility.Random( 0xECA, 9 ); // bone graphic
GumpID = 0x2A73;
DropSound = 0x48;
Hue = 0;
Weight = 5; // final, so we can bury our bodies
ProcessDelta();

Maybe consider what that change may result in.

You really need to just either run a debug session and set a breakpoint on Open() or start commenting out code and testing.
 
Last edited:
Finaltwist said:
'Greetings all,

we've had a mysterious lag issue in ultima adventures for some time now - whenever a player dies, the entire server will lag for some time. after some experiementing around the same was found to happen when a player looted his corpse. we dug in more and realized it was directly related to the number of items the player had when he died/when he loots. the more items in the pack, the longer the lag.

We did some looking around but can't find any cause for this in corpse.cs both in core of in server scripts. something is making the server lag whenever items are transfered to/from corpses. HAs anyone encountered this? you can get the entire server code by downloading the resource....

If anyone can help fix this, ill be incredibly grateful.. hell ill even add a statue of that person in the game, commemorating the heroic deeds of the person!!!
any help is appreciated.
Click to expand...

I have some questions about what the actual experience is like. When you open your own corpse and have say 100 items on it, do you see the corpse open, and can you watch as items are slowly moved from corpse to pack one by one? Or does it lag such that you don't see anything happen for several seconds then suddenly everything that was on the corpse is in your pack, and then the empty corpse opens? How long of a delay are we talking about here?
 
Or does it lag such that you don't see anything happen for several seconds then suddenly everything that was on the corpse is in your pack, and then the empty corpse opens?
Click to expand...
This... for 100 items the lag is small, say half a second- but we allow 200 items in a pack (which i agree could be decreased if the lag issue can't be fixed). at 200 items, the lag is 1-1.5seconds. the lag is server wide for anyone on.

You really need to just either run a debug session and set a breakpoint on Open() or start commenting out code and testing.
Click to expand...
yes we've done this - commenting out anything but the corpse.AddItem( item); does not affect the lag. We've commented out everything else (tombstones, processdelta(), gump changes, etc etc). It's my suspicion that this is not an issue in odyssey because its not noticeable. they restrict pack items to 125 and to my knowledge, has not been used in a 10+ population server. I actually merged the code with runuo2.6 to see what any differences might be, both corpse.cs and container.cs and im at a loss as to what it might be causing this.
Post automatically merged:

oh, and thanks for everyones help :)
 
some complained that wasn't enough! The hosting service that runs the server is powerful enough to be able to handle 200 items moving - it does much more on a regular basis with little CPU use (CPU averages 50% use). so i don't think its a resource thing. Obviously reducing it back to 125 is the fallback solution but id rather get to the bottom of the issue if i can right?
 
Thanks missed that, and had the same problem! lol
Post automatically merged:

Is there no override for AddItem in Container.cs?
Wouldn't we need Item.cs?
 
Last edited:
The only thing that really stands out to me, is this loop check (below) is being run in ServUO ahead of the one you have here, and is missing in 'Open' of Corpse.cs
Code: 
                    for (int k = 0; k < EquipItems.Count; ++k)
                    {
                        Item item2 = EquipItems[k];

                        if (!items.Contains(item2) && item2.IsChildOf(from.Backpack))
                        {
                            items.Add(item2);
                            gathered = true;
                        }
                    }

After that, it is possible it could be anywhere in OnAdded, at the end of AddItem in Item.cs.., but who knows maybe that will help.
Good luck!
 
Last edited:
ill have a look at onadded, but i merged the item.cs with runuo2.6 and didn't find any red flags. its an odd thing

thanks for having a look at least, friend!
 
Did you add the above checks to Corpse.cs?
Post automatically merged:

Before this..
Code: 
for ( int i = 0; !didntFit && i < items.Count; ++i )
{
Item item = items;
 
Last edited:
it might help to quote the actual checks instead of what comes after, so i don't have to load visual studio :)

do you mean the death robe check and the quest arrow check above for ( int i = 0; !didntFit && i < items.Count; ++i )
?

they are not the cause of the lag, the death arrow just points to a healer/ankh and the death robe just gets deleted. sorry not sure what checks you are refering to
 
I posted it, you must be looking past it on accident..

Code: 
                    for (int k = 0; k < EquipItems.Count; ++k)
                    {
                        Item item2 = EquipItems[k];

                        if (!items.Contains(item2) && item2.IsChildOf(from.Backpack))
                        {
                            items.Add(item2);
                            gathered = true;
                        }
                    }

needs to go before..
Code: 
for ( int i = 0; !didntFit && i < items.Count; ++i )
{
Item item = items;
...

If you open up ServUO Corpse.cs you will be able to see it in its entirety.
I downloaded this project a short while ago, there is no Server side source?
 
yes its the runuo2.2 folder in the utilities
Post automatically merged:

oooh... i see it now... when a player dies, the script is adding ALL the players items to the items list... youre saying it should only add items from equipitems
 
Last edited:
Greetings Servuo! We still have not been able to finx this issue... we know the following:

1. It is not CPU based - we recently double CPU cores on our server and it changed nothing.
2. Entire server lag happens when a player sells large amounts of items to an npc vendor (40+ items is 1-2 seconds)
3. Entire server lag happens when a player dies and items are moved from backpack to corpse
4. Entire server lag happens when a player loots corpse and entire corpse items get moved back to the player
5. Entire server lag happens when a player moves a bag with other bag who have other bags and a large number of items from the ground to backpack
6. Server does not lag when bag in 5. is moved from ground to ground
7. entier server lags when a player at a champ spawn kills 30-60 enemies in one hit (wither) and items are moved to corpses.

Thats all we have... I'm attaching the entire server files, with core scripts too, we are desperately asking the comunity for help as this problem has been plaguing us for ages, and no solution found...

latest server code
 
Here is a video i made re-producing the issue by simply moving a stack within a players pack over and over. The further from the players parent pack the item is "container in container" the worse the ping spikes become.
To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.
 
Would also need some more info. Since after making it work after fixing all the errors of missing dlls and all ;)

What kind of server are you running?
OS?
Version of the OS?
How many Cores?
Do you know the freq of the Cores?
How much RAM?

Just in case, HDD or SSD?
Does it happen when you are online alone?
Does it happen when you try it local, on your pc?
 
Windows hosted server windows NT, 4 cores, 4 gb ram (1.7 gm usage), cpu cores only used max 55%, SSD 2gb (only use a fraction of that),hard to test local because there has to be others online to notice the lag.

we used to have 2 cores, had the issue, doubled the cores, no change in server lag. on item move.

oh, and got a report that still exists offline.
 
Last edited:
Ok so it can happen offline as well.

Locally with just the files from the drive? (and making them work) I had like 8 spawners instantly spawning 1k Skeletons and nuking them earthquake. Not feeling any lag there.

May need to set up a VM or something for that idk
 
I just tested it in offline with netstate open And moving ore fast in a MinerPouch can get me a ping of 150+
Post automatically merged:

PyrO said:
Ok so it can happen offline as well.

Locally with just the files from the drive? (and making them work) I had like 8 spawners instantly spawning 1k Skeletons and nuking them earthquake. Not feeling any lag there.

May need to set up a VM or something for that idk
Click to expand...
Did you open the netstate from the client it can help to see server response time getting higher!
 
Tested it again, locally, 1 core, used the minerspouch and ore, no lag whatsoever.
---
If you find a way to reliably test it locally, then I will see to it.

Also not sure if it may be connected to your world save / item and mobile count
 
Cannot confirm yet but the bug seems to be related with the fact that item does not decay. Tested with the save from last month and with a wipe and that change everything at the latency.
 
PyrO said:
Tested it again, locally, 1 core, used the minerspouch and ore, no lag whatsoever.
---
If you find a way to reliably test it locally, then I will see to it.

Also not sure if it may be connected to your world save / item and mobile count
Click to expand...
Any idea on how to remove every item that are not locked down. I would like to test it as if the game as item decay to see if less movable object make a big change.
 
Why would moving an item from a corpse to a players backpack be affected by items not locked down throughout the server? Where would this be in the code?

also @PyrO , you can test this offline by having, say 300 items in your pack, dying and/or autolooting the corse with 300 items. I know we can reduce player packs to 125 items like most shards, but the point here is that the server shouldn't lag with 300 items being moved in the first place.
Post automatically merged:

also if above is true and that this is somehow related to the item count in the save, then the item lag won't happen unless you have the save from the server. ill attach it here

save file which will show the item lag.

i'd be curious why no item decay is causing this, and whether it could be fixed. the save has 800k items or so
 
Last edited:
Tested it again.
250 Items in the backpack, moving items quickly, no lag.

put them all in a separate bag, move the full bag around, no lag.

Cores are again limited to 1
 
Took the saves from adventures, where there was only 1 in the backups. Loaded it, deleted the zebras, because serialize error from the save.

logged in, tested, no lag. And I spam dragged quickly too
 
is there any way you could login to the server (login info is in the jpg in the adventures folder) and try there?
I haven't seen the item moving lag, just heard it from others above, which may be due to client side issues.

what I HAVE seen is the selling 60 items to san npc (e.g. tinker tools to a tinker npc), player dying with 200 items in pack, and looting corpse. players dying lags entire server for 1-3 seconds sometimes, for everyone logged in.

also, collecting 40-50 mobs at a champ spawn and doing a wither (killing all mobs at once) causes lag as well, verified and confirmed.
 
Logged in, when moving items (regardless of how many Items you have in your back, the ping can go up to 400, reported my the client.
cmd ping stays at 150 ms for me.

Teleporting does the same thing. As in moving in and out of the bank
 
You must log in or register to reply here.
Back