Send Custom Loggers to Splunk

[pyieh]

Background: Jenkins system log defaults to ignore all logs below the INFO priority level, meaning certain logs of interest aren't being sent to Splunk. Wanted a way to configure Jenkins so that certain logs are still being sent to Splunk regardless of log priority level (as this level is up to the plugin implementation and not controllable by the end user)

Adds a new section to the Splunk-Devops configuration page that allows Admins to specify any custom loggers to have their logs sent to Splunk regardless of log level.

Configuration page will have a dropdown selection that is populated by the list of configured custom loggers at <JENKINS_URL>/log/

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

[pyieh]

@fengxx Can I get a review on this?

[saper]

Background: Jenkins system log defaults to ignore all logs below the INFO priority level

I have a question here: does it help if one changes the log level

splunk-devops-plugin/splunk-devops/src/main/java/com/splunk/splunkjenkins/JdkSplunkLogHandler.java

Line 32 in b290886

setLevel(filterLevel);

and removes the check

splunk-devops-plugin/splunk-devops/src/main/java/com/splunk/splunkjenkins/JdkSplunkLogHandler.java

Line 53 in b290886

if (record.getLevel().intValue() < Level.INFO.intValue()) {

?

[pyieh]

@saper I believe we tried this and you can use it to send more logs to splunk, however, that may send all unwanted logs to Splunk. This feature would allow more granular handling of which logs to send.

Background: Jenkins system log defaults to ignore all logs below the INFO priority level

I have a question here: does it help if one changes the log level

splunk-devops-plugin/splunk-devops/src/main/java/com/splunk/splunkjenkins/JdkSplunkLogHandler.java

Line 32 in b290886

setLevel(filterLevel);

and removes the check

splunk-devops-plugin/splunk-devops/src/main/java/com/splunk/splunkjenkins/JdkSplunkLogHandler.java

Line 53 in b290886

if (record.getLevel().intValue() < Level.INFO.intValue()) {

?

[pyieh]

@fengxx Can I get a review for this PR ?

[fengxx]

Existing logic https://github.com/jenkinsci/splunk-devops-plugin/blob/master/splunk-devops/src/main/java/com/splunk/splunkjenkins/LoggingInitStep.java#L40
Logger.getLogger(rootLoggerName).addHandler(JdkSplunkLogHandler.LogHolder.LOG_HANDLER); registers to root logger, that means every log record will loop the config items. Assume you have 10 targets configured, and 1000 log records per second, that means 10000 additional checks even if the logger is not in the config list

[fengxx]

this depend on LogRecorderManager, which adds a https://github.com/jenkinsci/jenkins/blob/75b4b31efd3f3340805497b6b23ff4b0cb32d2f3/core/src/main/java/hudson/util/RingBufferLogHandler.java automatically, that means if you want to sent to splunk verbose level log, it has to be logged to RingBufferLogHandler too, unnecessary performance burden.

[fengxx]

Thanks, but it will bring kind of performance issue. IMHO, it can be archived in other ways, as you know, Jenkins support init.groovy.d, you can add a setup_debug_log.groovy file there with similar content

import java.util.logging.*
import com.splunk.splunkjenkins.JdkSplunkLogHandler

def splunkFineHandler = new JdkSplunkLogHandler()
splunkFineHandler.setLevel(Level.FINE)
["com.foo","com.bar"].each { name ->
	def logger = Logger.getLogger(name)
        logger.setLevel(Level.FINE)
	//remove INFo handler and replace it with FINE handler
	logger.getHandlers().each { handler -> if (handler instanceof JdkSplunkLogHandler) logger.removeHandler(handler) }
	logger.addHandler(splunkFineHandler)
}
println "installed splunk FINE log handler"

of course you can adjust it dynamically on Jenkins script console too