[1.10] [FTB Beyond] Getting seemingly random server stalls with NBT Tags

[john01dav]

I am currently running...

• SpongeForge version: 1.10.2-2281-5.2.0-BETA-2353 (This is not the latest because I have far too many plugins to be on top of updates for all of them. I will update SpongeForge now however, and reply back here if this occurs again. In the mean time I want to get it posted so if anyone knows what's causing it it can be fixed.)
• Forge version: 12.18.3.2281
• Java version: build 1.8.0_131-b11 (Oracle, not OpenJDK)
• Operating System: Ubuntu Server 16.04

• Plugins/Mods: Plugins (189): Minecraft, Minecraft Coder Pack, Forge Mod Loader, Minecraft Forge, SpongeAPI, SpongeForge, Applied Energistics 2 Core, Aroma1997Core, Not Enough Items, OpenComputers (Core), CoFH ASM, FoamFixCore, ccl-entityhook, CoFH Core, Actually Additions, NuVotifier, ActuallyWorkingVoteListener, IndustrialCraft 2, Advanced Machines, Advanced Solar Panels, Applied Energistics 2, BD Lib, AE2 Stuff, InfinityLib, AgriCraft, Baubles, Botania, Akashic Tome, Just Enough Items, AppleSkin, Aquaculture, ArchitectureCraft, Aroma1997Core|Helper, AutoBroadcaster, Morph-o-Tool, Psi, Quark, AutoRegLib, Bed Bugs, Better Builder's Wands, BiblioCraft, Guide-API, Blood Magic: Alchemical Wizardry, Blood Arsenal, Bookshelf, CodeChicken Lib, Brandon's Core, Ceramics, Chameleon, Chest Transporter, Chisel, MCMultiPart, Chisels & Bits, ShadowMC, Clipboard, CodeChicken Core, Compact Solar Arrays, Cooking for Blockheads, CraftTweaker JEI Support, MineTweaker 3, Dark Utilities, PTRModelLib, Decocraft, Forestry, ElecCore, EnderStorage, OpenComputers, DeepResonance, Discord Chat, Draconic Evolution, Storage Drawers, Drawers & Bits, Elevator Mod, Embers, EnderCore, Ender IO, Engineers Workshop, Valkyrie Lib, Environmental Tech, ET Lunar, Extra Utilities 2, Zero CORE, Extreme Reactors, Flat Colored Blocks, FlexibleLogin, SonarCore, FluxNetworks, FoamFix, Forgelin, Forge Multipart CBE, Forge Microblocks, Minecraft Multipart Plugin, FTBLib, Funky Locomotion, Funnels, GenDustry, Gravestone, Gravitation Suite, GriefPrevention, Hammer Core, HelpFixer, Hopper Ducts, Mantle, Tinkers' Construct, The One Probe, Immersive Engineering, LunatriusCore, InGame Info XML, Interaction Wheel, Inventory Tweaks, Iron Backpacks, Iron Chest, JEI Bees, JourneyMap, KleeSlabs, Long Fall Boots, LuckPerms, MalisisCore, MalisisDoors, Mercurius, Modular ForceField System, MinecraftMarket, MMCCChunkLoader, MMCCRestarter, MmmMmmMmmMmm, mob_grinding_utils, MTLib, Mod Tweaker, Railcraft, More Bees, ModPack Utilities, ModPack Basic Tools, MrTJPCore, ShetiPhian-Core, Multi-Storage, Natura, Nether Portal Fix, Nucleus, Open Glider, p455w0rd's Library, Pam's HarvestCraft, Prism, ProjectRed Core, ProjectRed integration, ProjectRed Transmission, ProjectRed Illumination, ProjectRed Expansion, ProjectRed Transportation, ProjectRed Exploration, RebornCore, QuantumStorage, Ranged Pumps, reborncore-MCMultiPart, Thermal Foundation, Redstone Arsenal, Refined Relocation, Refined Storage, Reliquary, RFTools, RFTools Control, RFTools Dimensions, Roots, Sampler, Signals, Simple Retrogen, Simply Jetpacks 2, Sleeping Bag, Snad, Solar Flux Reborn, Steve's Carts 2, Storage Drawers Extras, Super Circuit Maker, Super Crafting Frame, TechReborn, Thermal Expansion, TickProfiler, Tinkers Tool Leveling, TOP Addons, TorchMaster, Translocator, Useful Nullifiers, WanionLib, Wireless Crafting Grid, Wireless Crafting Terminal 2, Woot, WirelessRedstone-CBE, XNet, YABBA, UniDict

What is the expected result?
I expect Sponge to run and not stall.

What is the current result?
Sponge stalls randomly. There are hs_err_pid logs in my server's root, and they say that the JVM is running out of memory. WarmRoast also can't connect to the JVM to see what it's doing. As such, this looks like the JVM is completely running out of memory and thus stopping. As I rarely get past 10 players and have 15GBs allocated (it doesn't get past ~7GBs under the most active of normal use), I doubt allocating more ram is the answer. I believe this is a sponge issue since the stall report says that the server is doing NBT stuff, and quite a lot of Sponge is in the stack trace too. If you think this would be better reporter somewhere else, please tell me where that is.

Server Logs
Sponge/Forge/etc. seem to place logs all over. I got what I can here, let me know if yuo need more:
https://pastebin.com/R1fugKY6 (uncompresed .log.gz file)
https://pastebin.com/KQM1g2uF (hs_err_pid)
https://pastebin.com/LLWbsaJe (stall report)

[Niko-sk2x]

(note: I know nothing about the stalls, I just know what causes the crashes)

You are right that allocating more RAM is not the answer. But you would be surprised to see that allocating less ram is the answer to the crashes. You allocated a lot of RAM to it, but your system doesn't have that much physical RAM. Most of the time not all the RAM you specified is used, but when it is needed - the system can't allocate that much and it crashes. Allocate less RAM to java and it will work. And just in case - you can add some swap space (equivalent of windows pagefile).

[john01dav]

The system has 32 gb's of ram total. The java arguments tell the jvm (via xms) to allocate all 15 gbs when the server starts. Htop conforms that this happens. The ram allocations of all servers on this machine add up to 23 gbs, which leaves 9 gbs free on the system. During normal usage most of these 9 gbs are free.

…

On Jun 19, 2017 8:49 AM, "Bartosz Skrzypczak" ***@***.***> wrote: (note: I know nothing about the stalls, I just know what causes the crashes) You are right that allocating more RAM is *not* the answer. But you would be surprised to see that allocating *less* ram is the answer to the crashes. You allocated a lot of RAM to it, but your system *doesn't* have that much physical RAM. Most of the time not all the RAM you specified is used, but when it is needed - the system can't allocate that much and it crashes. Allocate less RAM to java and it will work. And just in case - you can add some swap space (equivalent of windows pagefile). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1595 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADzEodLlkb6Eqddp5KyjAVoY2jzC2Bgdks5sFnxNgaJpZM4N9svN> .

[Deamon5550]

A random stall report is never going to tell you anything useful as its just what the server happened to be doing at the time but doesn't mean anything about what the server has spent 60 seconds doing. You're going to need to either use timings/warmroast/a profiler to dee that your issue is actually.

Sponge definitely has nothing to do with your server being unable to allocate memory, but it is correct that reducing the memory may definitely help (and you should check your GC flags as well to make sure you're using the G1GC). A profiler can take a memory snapshot that will help you debug that as well.

[john01dav]

I've had this issue multiple times, and the stack trace in the profiler is always the same. As such, it looks like some sort of nbt stuff is causing the server to run out of memory. Perhaps a packet similar to a zip bomb ( https://en.m.wikipedia.org/wiki/Zip_bomb) is getting sent (most likely an out of control loop bug in a mod)?

…

On Jun 19, 2017 10:57 AM, "Nick Condé" ***@***.***> wrote: A random stall report is never going to tell you anything useful as its just what the server happened to be doing at the time. You're going to need to either use timings/warmroast/a profiler to dee that your issue is actually. Sponge definitely has nothing to do with your server being unable to allocate memory, but it is correct that reducing the memory may definitely help (and you should check your GC flags as well to make sure you're using the G1GC). A profiler can take a memory snapshot that will help you debug that as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1595 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADzEodiBIkQ6k3JfFHou-0h3tMvlc-lOks5sFppvgaJpZM4N9svN> .

[Deamon5550]

Then attach a profiler and show us the results.

[phit]

Seriously lower the RAM allocation, 6GB min and max are plenty for beyond, especially with your player counts..

As I rarely get past 10 players

Here are some arguments I can recommend, based on this Reddit post and my own experience

-XX:+UseG1GC -Xmx6G -Xms6G -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=50 -XX:G1HeapRegionSize=32M

[john01dav]

@Deamon5550 I tried to attach WarmRoast, but by the time the issue's symptoms (in-game freeze) become available, the VM is already too dead for WarmRoast to get any data. It times out when it makes it's connection to the VM.

[bloodmc]

@john01dav unless you update SF to latest, this will be ignored. Update and provide new logs. Also remember you can always dump the current memory by using /sponge heap. You can then analyze the heap dump with a profiler.

[john01dav]

I'm already updated, and I am waiting for it to happen again. It's very intermittent.

…

On Jun 19, 2017 11:24 PM, "bloodmc" ***@***.***> wrote: @john01dav <https://github.com/john01dav> unless you update SF to latest, this will be ignored. Update and provide new logs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1595 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADzEoW3kque97o6G7wJEoI9rWFGQAhBQks5sF0lhgaJpZM4N9svN> .

[gabizou]

Something that I've seen with another server is TickProfiler being installed on the server can mean that certain major systems in Sponge are not functioning as they should (and are likely going to spam the hell out of the server). Do you have a server log with cause tracker verbosity turned on?

[phit]

might be related to rwtema/FunkyLocomotion#104

[Zidane]

Closing as this isn't a Sponge issue. I'd take a look @phit 's comment and go from there (if you still use that mod).