Memory Leak - traffic_ctl config reload

[alxtv]

Hi,

I have noticed that the trafficserver's memory usage is increasing with each config reload. After each config reload it's allocating ~200MB RAM until the system has free memory and then dies (oom killer or trafficserver cannot allocate memory). There are around 7700 remap rules and 1800 SSL certificates on the server if that matters.

Tested on - Apache Traffic Server - traffic_server - 7.1.5 - (build # 110507 on Nov 5 2018 at 07:03:44)
executing - touch /etc/trafficserver/remap.config && touch /etc/trafficserver/ssl_multicert.config && traffic_ctl config reload
But older version 7.1.4 has the same issue.
Edit:
I have noticed that it's allocating more memory while reloading the ssl_multicert.config

Memory Dump
ts-start.txt
ts-after-few-reloads.txt

edit 2:
Even with empty remap.config and no plugins activated, touch /etc/trafficserver/ssl_multicert.config && traffic_ctl config reload is increasing the memory usage.

Edit 3:
Empty ssl_multicert.config and the memory leak is completely gone.

[randall]

I just went through and found/fixed a number of leaks related to reloading but nothing really with ssl_multicert. There was a fix for when a cert didn't have an OCSP URL.

Which version of 7.1.5 were you using here?

Any more details about the contents of ssl_multicert.config? What's the typical config line look like? Just trying to get an idea of the features/codepaths used here.

[alxtv]

The configuration of ssl_multicert.config is very simple it's like:
ssl_cert_name=/home/ssl/domain1.com.cert ssl_key_name=/home/ssl/domain1.com.key ssl_ca_name=/home/ssl/domain1.com.ca
ssl_cert_name=/home/ssl/domain2.com.cert ssl_key_name=/home/ssl/domain2.com.key ssl_ca_name=/home/ssl/domain2.com.ca
ssl_cert_name=/home/ssl/domain3.com.cert ssl_key_name=/home/ssl/domain3.com.key ssl_ca_name=/home/ssl/domain3.com.ca
ssl_cert_name=/home/ssl/domain4.com.cert ssl_key_name=/home/ssl/domain4.com.key ssl_ca_name=/home/ssl/domain4.com.ca
ssl_cert_name=/home/ssl/domain5.com.cert ssl_key_name=/home/ssl/domain5.com.key ssl_ca_name=/home/ssl/domain5.com.ca
.............
ssl_cert_name=/home/ssl/domain1900.com.cert ssl_key_name=/home/ssl/domain1900.com.key ssl_ca_name=/home/ssl/domain1900.com.ca
............
OCSP is deactivated and all the configuration is almost default. Nothing changed especially for the SSLs.

Source downloaded on 5th Nov from branch 7.1.x so it should include your memory leak fixes.

I have tried with older version 5.3.0 (I think) got from epel repo for CentOS 7 but still has the same issue. There might be also a memory leak while reloading the remap.config but leaking less memory than the ssl_multicert.config. Will try that later.

EDIT:
I think that this memory leak may be related to the openssl used for building the Traffic Server. I have downloaded openssl-1.0.2p, build TS with it and the leak is gone for now. I am still testing it and will share the results later. The version that leaked memory has been built with the CentOS openssl 1.0.2k installed from the base CentOS repo.

[randall]

That would explain why I don't see similar leaks. I run with OpenSSL 1.1.x (currently 1.1.0i)

[alxtv]

Definitely the leak was caused by the OpenSSL. Running for 24h on multiple instances and haven't noticed any memory leaks. I am not sure what's the case with the CentOS openssl but seems very strange.

[zwoop]

Can we close this ?

[alxtv]

Yes.

[alxtv]

I've faced another memory leaks.
Tested with version (Apache Traffic Server - traffic_server - 7.1.5 - (build # 120309 on Dec 3 2018 at 09:41:40)) built with openssl-1.0.2p.

The case now is that it's leaking more memory while reloading the remap.config ~(150-200MB) after each config reload.
There's one strange thing - the memory dump doesn't show any increase of the allocated memory after reloading the ssl configuration but actually it was allocated additional 700MB to the initial ~350MB.

There are ~4000 ssl certificates loaded by the server and ~18000 remap rules.

reload-ssl.txt
reload-both.configs.txt
reload-remap.txt

Any suggestions ?

EDIT: It correlates directly to the number of remap rules, more remap rules = more memory allocation after 'traffic_ctl config reload'...

EDIT 2: I can confirm that this is 100% related to the count of the remap rules, tested with almost 40000 random generated remap rules and the memory allocation after 'traffic_ctl config reload' has doubled compared to ~18000 remap rules.
I know that it's normal for the server to allocate more memory for more remap rules but I don't think that it's normal to allocate more and more memory after each config reload.

EDIT 3: According to my tests, this is almost fixed in 8.0.0 version. What I mean by "almost" is that when i run the server with ~35000-40000 random generated remap rules, it's starting with ~450MB RAM, after first "config reload", it's doubling the allocated memory to ~850MB, after third "config reload" it's allocating more ~150MBs up to ~960MB and then no matter how many reloads I make, it does not allocate any more RAM. I am reloading the configuration with:
touch /etc/trafficserver/remap.config && touch /etc/trafficserver/ssl_multicert.config && /usr/bin/traffic_ctl config reload

While the same test on 7.1.5 with just reloading the remap.config by executing touch /etc/trafficserver/remap.config && /usr/bin/traffic_ctl config reload is allocating ~400MB RAM after each config reload.

[alxtv]

Hello again,

I made a test with clear default ATS installation (and configuration). I have added 100k random simple remap rules (map https://domain.com/ https://127.0.2.54/ - nothing else), no plugins, no ssl certificates.
After starting ATS, it allocates ~1G RAM, after touching remap.config and running "traffic_ctl config reload" it allocates additional 1GB ram, and more ~300MB on the second "traffic_ctl config reload". After that I am reloading the config with "traffic_ctl config reload" more than 10 times but it doesn't allocate any more memory.

I have noticed that while starting the server it executes "mmap(NULL, 270336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1b2337d000" - 1664 times (with different memory addresses of course) while loading the remap.config (100k remap rules). That's about ~430MB memory from total 1GB which is probably okay for the amount of remap rules.

When I touch the remap.config and run "traffic_ctl config reload", the ET_TASK thread executes "mprotect(0x7f1afe7aa000, 4096, PROT_READ|PROT_WRITE) = 0" - 58133 times - (~230MB) and "mprotect(0x7f1b0227b000, 266240, PROT_READ|PROT_WRITE) = 0" - 3092 times - (~785MB) which is exactly the amount ATS allocates after first "traffic_ctl config reload".

For the second touch of remap.config and running "traffic_ctl config reload", the ET_TASK thread executes "mprotect(0x7f1afe7aa000, 4096, PROT_READ|PROT_WRITE) = 0" - 61371 times - (~240MB) which is not exactly what I mentioned earlier (~300MB) but I am not getting the exact values in bytes so it should be the same.

When I touch the remap.config and run "traffic_ctl config reload" for the third time, the ET_TASK thread is not executing mprotect or mmap anymore, and no more memory is allocated. I have tried to touch and reload the config many times in a row but no more memory is being allocated.

This was tested on latest version from the master branch on Linux 4.18.0-13-generic #14-Ubuntu SMP 2018 x86_64 and confirmed on CentOS 7 with ATS version 7.1.5.

I am not sure why it's working this way and if it's okay or not. It just looks strange for me.

I am attaching the strace logs of all ATS threads in case anyone wants to see them.

strace.zip

Let me know if you need any additional information on the case.