Log Monitoring Tool in Python

  • Python
  • Pandas
  • Sendgrid for Email notifications
  • More fuzzy search and not a direct keyword search
  • Other notification channels like SMTP email support, Slack channels
  • More log type support like other format of logs, logs in clod locations like S3

How to use

Pre-Requisites

  • Python
  • Install all packages from requirements.txt
  • Virtual environment
  • An API Key from sendgrid (https://sendgrid.com)

Usage

  • emailaddress: Comma separated email Ids to which it will send notifications
  • keywords: A list of keywords which the application will look in the logs to identify errors. If any of these keywords are present in any of the logs, the application will report it as a failure
  • smtphost,smtpuser,smtppwd: Reserved for a future enhancement to support SMTP settings
  • from: The email notifications will show this as the from email
  • subject,body: Email Subject and Body content of the error email notifications sent out
  • numberofjobs: Number of applications whose logs it will monitor
  • logmodifiedduration(minutes): The application will only check for logs which are modified within the minutes mentioned in this parameter. This will ensure it only picks up latest log files and not everything.
  • lastrunflag: Keep the values as Y
  • logtype: This is the type of log which it will work on. For now it supports csv and txt. Specifying either of them will enable the application to monitor all logs which are in text format.
  • wholewordsearch: Keep this as Y. Placeholder for a future enhancement to support fuzzy search
  • sendmode: There are two modes supported by the application:
  • Summary: It will send summary notifications each time it is executed. The application will search all the logs specified in the path locations and send out a summary for each log its monitoring.
  • Error: It will only send out email notification when it finds a match for the keywords in the monitored logs
  • sendattachments: This parameter controls whether the log file is sent as an attachment to the email or not. Based on the selection, the monitored log file is attached to the email notification which is sent out.Allowed values- Y or N.
  • summarysubject, summarybody: Email subject and body content when Summary is selected as the mode.
  • sendgrid_api: The api key for the sendgrid account which the tool will use to send out emails
  • Create virtual environment
    python -m venv log_monitor_env
  • Install dependencies
    pip install -r requirement.txt
  • Run the app
    python main.py
Email

Example Usage patterns

  • Run this application after executing any scheduled or cron task so its log can be checked for errors from the latest execution
  • Include this in a Devops pipeline as as a step to monitor logs from various pipeline phases
  • Monitor web server logs and schedule this app to run periodically and check server logs for errors

Technical details

  • The program reads all the input parameters and the paths from the resources folder
  • Based on the config settings, a log object is initialized from the logfile class. This calls the method to read the log and perform the keyword search on each of the log files
  • Once the log files are scanned, an email object is instantiated from the email communications class
  • The email object takes the output from the log scan and builds the email body based on the settings from the config file
  • Based on the send mode selected in the config, separate methods are called to prepare different email content
  • Once the email body is complete(based on Summary or Error selected in the config), it calls the SendGrid API to send out the necessary emails. It uses the API Key specified in the config
  • All the steps performed in the code are logged in log files in the logs folder. These are logs created by the loggerclass.

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Amlan Chakladar

Amlan Chakladar

Cloud architect and DevOps engineer. Love to code and develop new stuff. A nerd by nature.