Python Script to Read Logs From Sumo Logic

Notation:

• This tutorial requires access to Oracle Cloud. To sign up for a free account, run across Go started with Oracle Cloud Infrastructure Free Tier.
• It uses case values for Oracle Deject Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud surroundings.

Move logs from Oracle Cloud Infrastructure into Sumo Logic

Introduction

The Oracle Cloud Observability and Manageability platform aims to meet our customers where they are. We understand that they have standardized their operational postures with popular third-political party observability tools and we desire to exist interoperable with those tools then our customers tin can continue using the tools they have invested in with Oracle Deject Infrastructure (OCI).

In this tutorial, nosotros will walk y'all through how you can move logs from OCI into Sumo Logic. Sumo Logic is a pop observability tool that provides monitoring and security services that provide total visibility into your applications.

Our solution compages at a high level is as shown beneath:

Create a Custom HTTP Source Collector in Sumo Logic

In your Sumo Logic account, yous need to create an HTTP custom collector app equally described in the steps below.

1. Click Setup Wizard.

   

2. Click First streaming data to Sumo Logic.

   

3. Click Your Custom App.

   

4. Click HTTPS Source.

   

5. Configure your HTTP source equally shown below.

   Notation: The HTTP source is a metadata tag, stored with your ingested logs, and is useful when searching and filtering logs afterward in Sumo Logic. Each log line nosotros are going ingest will get-go with a timestamp of its issue occurrence so select the option Use fourth dimension zone from log file.

   

   As you move to the next screen, we get the HTTPS endpoint for our logs to upload from OCI, using a Mail service HTTP call. Delight take a notation of this endpoint as we volition configure our function to use this endpoint to upload logs to Sumo Logic.

   

Configure the Logs You Want to Capture

You tin set whatever logs as input for Service Connector Hub and hence ingest into Sumo Logic. For simplifying this tutorial, nosotros will capture Oracle Cloud Infrastructure (OCI) generated logs for write-events to an arbitrary saucepan of your pick.

1. In the Oracle Cloud Panel, click the navigation bill of fare, select Logging, and and so select Log Groups.

   

2. To create a log group, click Create Log Group.

   

3. Select your compartment, add LogGroupForBucketActivity for the name and add together a description. Click Create.

   

4. Select Logs from the Logging menu. Y'all will see a screen similar to below.

   

5. Click Enable service log and enter the following information:

   • Service: Select Object Storage
   • Resources: Choose an arbitrary saucepan(for example, BucketForSumoLogic) that you would like observed with the logs.
   • Log Category: Select Write Access Events
   • Log Name: Enter a name for your log, for example, logForBucketActivity.
   • Log Grouping: Select the LogGroupForBucketActivity log grouping for the log that you just created in the previous step

6. Click Enable Log.

   

   At present every time a object is uploaded to the BucketForSumoLogic bucket,a log entry will be added to the logForBucketActivity log.

Configure Oracle Functions for Ingesting Logs into Sumo Logic

1. In the Oracle Cloud Panel, click the navigation menu and select Solution and Platform. Select Functions under the Developer Services carte.

2. Click Create Application and enter a name, for instance, SumoLogicFnApp.

   

3. Once you create the awarding, click your awarding name and select Getting Started from the Resources bill of fare.

   

4. Launch Cloud Trounce.

5. Use the context for your region.

                 fn listing context fn use context us-ashburn-1                          

6. Update the context with the function's compartment ID.

                 fn update context oracle.compartment-id <compartment-id>                          

7. Update the context with the location of the registry you want to use.

                 fn update context registry iad.ocir.io/<tenancy_name>/[YOUR-OCIR-REPO]                          

   Replace iad with the three-digit region code for your region.

8. Assuming you have created the Auth Token already, log in to the registry using the Auth Token equally your password.

                 docker login iad.ocir.io                          

   Replace iad with the iii-digit region lawmaking for your region.

   You are prompted for the following information:

   • Username: <tenancyname>/<username>
   • Password: Create a password

   Note: If you lot are using Oracle Identity Deject Service, your username is <tenancyname>/oracleidentitycloudservice/<username>.

   Verify your setup by listing applications in the compartment.

9. Generate a 'hello-world' boilerplate part.

                 fn init --runtime python sumologicfn                          

   The fn init control will generate a folder called SumoLogicfn with 3 files inside: func.py, func.yaml, and requirements.txt.

   Open func.py and replace the content of the file with the following lawmaking.

   1. Import the necessary Python modules, as shown in the following snippet.

                        import io import json import logging import os  import requests from fdk import response                                  

   2. Ascertain a function to parse the log information and invoke the Sumo Logic API to ingest the logs.

                        # This method is the entrypoint for your Part invokation  # aka the method invoked past the OCI Fn platform # it will receive the list of log entries from OCI as input in the form of bytestream # the method proper noun will be defined in func.yml def handler(ctx, data: io.BytesIO = None):     logger = logging.getLogger()     logger.info("function start")      # Sumologic endpoint URL to upload OCI logs to HTTP custom app.     # this value will be divers divers in func.yaml     sumologic_endpoint = bone.environ['SUMOLOGIC_ENDPOINT']                                  

      For information most the format of the logs generated by the Oracle Cloud Infrastructure Logging service, run into Logging Format Overview.

   3. Call up the log entries from the Service Connector Hub received by our sumologicfn office as its invocation payload. Loop through these log-entries and log-lines 1 by one.

                        endeavor:     logentries = json.loads(data.getvalue()) # deserialize the bytesstream input as JSON array     if not isinstance(logentries, list):         logger.error('Invalid connector payload. No log queries detected')         raise      # Optional...log the input to the function as homo readble JSON.      # Not to be used in product     logger.info("json input from SCH")     logger.info(data.getvalue())       for logEntry in logentries:          logger.info("Extracting/Parse log details from the log entry json")         event_name = logEntry["data"]["requestResourcePath"] + '\t'         time_of_event = logEntry["time"] + '\t'         cmpt_name = logEntry["data"]["compartmentName"] + '\t'         bucket_namespace = logEntry["data"]["namespaceName"] + '\t'         bucket_name = logEntry["data"]["bucketName"] + '\t'         request_action = logEntry["data"]["requestAction"]          log_line = time_of_event + event_name + cmpt_name + \                     bucket_namespace + bucket_name + request_action          # Call the Sumologic with the payload and ingest the OCI logs         headers = {'Content-type': 'text/plainly'}         response_from_sumologic = requests.post(sumologic_endpoint,                                                 data=log_line,                                                 headers=headers)         logging.getLogger().info(response_from_sumologic.text)      logger.info("function end")     return  except Exception as e:      logger.error("Failure in the function: {}".format(str(e)))      enhance                                  

10. Replace func.yml contents as follows. Make sure you put the value for your SumoLogic_ENDPOINT that nosotros got in the previous step.

                                      schema_version                  :                  20180708                  proper noun                  :                  sumologicfn                  version                  :                  0.0.1                  runtime                  :                  python                  entrypoint                  :                  /python/bin/fdk /part/func.py handler                  memory                  :                  1024                  timeout                  :                  120                  config                  :                  SUMOLOGIC_ENDPOINT                  :                  [                  YOUR SUMOLOGIC API ENDPOINT URL Hither                  ]                              

11. Replace requirements.txt contents as follows.

12. Deploy your office.

                    fn -5 deploy --app sumologicFnApp --no-bump                              

13. Optionally, you lot can test your SumoLogicfn function with example input equally follows:

                  curl -O https://raw.githubusercontent.com/mayur-oci/sumologicfn/main/instance.json fn invoke sumologicFnApp sumologicfn < example.json                          

Create a Service Connector for Reading Logs from Logging and Triggering the Office

1. In the Oracle Cloud Console, click the navigation menu, and select Solution and Platform. Select Service Connectors under the Logging menu.

2. Click Create Connector, and from the Source drib-downwardly list, select Logging and from the Functions drop-downward listing, select Target.

3. On Configure Source Connexion, select your compartment proper name, your LogGroupForBucketActivity log group, and your logForBucketActivity logs.

4. If you want to utilise audit logs, click +Some other log, cull your compartment and add _Audit for Log Group.

   

5. If prompted to create a policy for writing to Functions, click Create.

The Service Connector is at present fix upwardly and volition trigger the office to ingest logs into Sumo Logic every fourth dimension it finds logs in the Logging service.

Visualize Oracle Cloud Infrastructure Logs in Sumo Logic

1. In Sumo Logic, select the Source - Custom App carte du jour to run into logs ingested from Oracle Deject Infrastructure (OCI) using our SumoLogicfn part.

   

Troubleshoot

This section shows how you lot tin can use a unproblematic email warning to monitor the condition of your solution.

For more details, see Overview of Functions.

Create a Topic and a Subscription for the Notification Service

1. In the Oracle Deject Panel, from the navigation menu in the upper-left corner, select Application Integration, and then select Notifications.

2. Click Create Topic and create a topic with the name my_function_status.

3. Cull your topic, click Create Subscription and use the post-obit example:

   • Protocol: Email and add together create a subscription with your e-mail.

4. The subscription will be created in "Awaiting" status. You will receive a confirmation electronic mail and will demand to click the link in the electronic mail to ostend your electronic mail address.

Bank check Metrics and Create an Alarm Definition from Metrics

1. From the navigation bill of fare in the upper-left corner, select Programmer Services, and then select Functions.

2. Choose the application and the function that y'all want to monitor.

3. From the Metrics page, get to the Functions Errors chart, click Options, and then click Create an Alarm on this Query.

4. Add a proper noun and under Notification, select Destination service as the notification service, select your_compartment, and then select Topic as my_function_status.

Monitor the Status Service Connector Hub

This section shows how yous tin can use a simple email alarm to monitor the condition of your Service Connector Hub (SCH).

For more details, refer to Service Connector Hub Overview.

Create a Topic and a Subscription for the Notification Service

1. From the navigation bill of fare in the upper-left corner, select Application Integration, and then select Notifications.

2. Click Create Topic and create a topic with my_sch_status name.

3. Choose your topic, click Create Subscription and employ the following example:

   • Protocol: Email and add create a subscription with your electronic mail

4. The subscription will be created in "Pending" status. Y'all will receive a confirmation email and volition demand to click the link in the email to ostend your email accost.

Check Metrics and Create an Alarm Definition from Metrics

1. From the navigation card in the upper-left corner, select Logging, and and so select Service Connectors.

2. Choose the connector that you want to monitor and from the Resource list in the left navigation panel, select Metrics.

3. From the metrics chart that you desire to add together the alarm to, for instance, "Service Connector Hub Errors", click Options and Create an Warning on this Query.

4. Add together a name and under Notification, select Destination service as the notification service, select your_compartment, then select Topic as my_sch_status.

Conclusion

This tutorial showed how Oracle Cloud Infrastructure and Sumo Logic customers tin can configure a highly scalable solution with low overhead for moving logs from Oracle Cloud Infrastructure Logging to Sumo Logic using Service Connector Hub and Oracle Functions.

Acknowledgements

• Author - Mayur Raleraskar, Solutions Architect

More than Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube aqueduct. Additionally, visit pedagogy.oracle.com/learning-explorer to get an Oracle Learning Explorer.

For production documentation, visit Oracle Help Center.

---

Move logs from Oracle Cloud Infrastructure into Sumo Logic

F41581-03

November 2021

Copyright © 2021, Oracle and/or its affiliates.

Python Script to Read Logs From Sumo Logic

Source: https://docs.oracle.com/en/learn/blog_sumologic/index.html

Share this post