diff --git a/docs/source/deploy_ui/start_using.rst b/docs/source/deploy_ui/start_using.rst index 9361a0e..d3520c2 100644 --- a/docs/source/deploy_ui/start_using.rst +++ b/docs/source/deploy_ui/start_using.rst @@ -26,6 +26,9 @@ Start using Cypienta UI .. image:: resources/home_page.png :alt: Home page :align: center + + .. note:: + The default ``Username`` is ``maestro`` and the default ``Password`` is ``changemenow`` How to use the Hide feature for events in UI diff --git a/docs/source/elastic/elastic.rst b/docs/source/elastic/elastic.rst index a994b13..a816893 100644 --- a/docs/source/elastic/elastic.rst +++ b/docs/source/elastic/elastic.rst @@ -1,6 +1,11 @@ Configure Elastic ================= +Prerequisites +------------- + +Make sure that you have deployed the Cypienta application detailed in :doc:`../getting_started/deploy` before integrating. + Logstash pipeline from Elastic Search to AWS S3 ----------------------------------------------- diff --git a/docs/source/getting_started/deploy.rst b/docs/source/getting_started/deploy.rst index 2431013..290e2e1 100644 --- a/docs/source/getting_started/deploy.rst +++ b/docs/source/getting_started/deploy.rst @@ -1,17 +1,40 @@ AWS Deployment ============== -.. _setup_lambda_repository: +.. _setup_lambda_repository_single_command: -Setup Lambda repository +Setup Lambda repository - Single quick command ----------------------- 1. Navigate to the AWS console, and select ``CloudShell`` at the bottom left of the console. Open the cloud shell in the region you want to deploy. +2. The following command will download shell script to setup lambda repository and output an ECR Image URI. Make note of the ECR Image URI to use in CloudFormation template. + + .. code-block:: shell + + $ wget https://github.com/cypienta/AWS/raw/v0.7/vrl-lambda.sh && sh vrl-lambda.sh + + .. note:: + Move to the section :ref:`setup_lambda_repository` for manual detailed steps. + +3. Once you make note of the ECR image URI for the VRL lambda, move to the section :ref:`deploy_cloud_formation` + +.. _setup_lambda_repository: + +Setup Lambda repository - Detailed manual steps +----------------------- + +1. Navigate to the AWS console, and select ``CloudShell`` at the bottom left of the console. Open the cloud shell in the region you want to deploy. You may run the following one line command, or continue to next step to run individual commands. + + .. code-block:: shell + + $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) && export REPO_NAME="cypienta-vrl-lambda" && docker pull public.ecr.aws/p2d2x2s3/cypienta/vrl-lambda:v0.1 && aws ecr create-repository --repository-name ${REPO_NAME} && export ECR_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com" && aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_URI} && docker tag public.ecr.aws/p2d2x2s3/cypienta/vrl-lambda:v0.1 ${ECR_URI}/${REPO_NAME}:v0.1 && docker push ${ECR_URI}/${REPO_NAME}:v0.1 && echo ${ECR_URI}/${REPO_NAME}:v0.1 + 2. Store the AWS Account ID, and ECR repository name to environment variable in cloud shell. .. code-block:: shell + # Save AWS Account ID as environment variable $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) # Replace value with ECR repository name you want to give @@ -39,9 +62,13 @@ Setup Lambda repository .. code-block:: shell + # Create ECR URI for ECR repository $ export ECR_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com" + # Login to the ECR repository $ aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_URI} + # Tag pulled image to push to ECR repository $ docker tag public.ecr.aws/p2d2x2s3/cypienta/vrl-lambda:v0.1 ${ECR_URI}/${REPO_NAME}:v0.1 + # Push the image to ECR repository $ docker push ${ECR_URI}/${REPO_NAME}:v0.1 7. Copy the ECR Image URI and make a note of it to use in CloudFormation template @@ -50,21 +77,28 @@ Setup Lambda repository $ echo ${ECR_URI}/${REPO_NAME}:v0.1 +8. Once you make note of the ECR image URI for the VRL lambda, move to the section :ref:`deploy_cloud_formation` + +.. _deploy_cloud_formation: Deploy resources using the Cloud Formation template --------------------------------------------------- -1. Clone the Github repo +1. On your local machine, download the template file from Github. `Template file `__. Or, use the following command to download the ``template.yaml`` file. .. code-block:: shell - $ git clone -b v0.7 https://github.com/cypienta/Lambda.git + $ wget https://github.com/cypienta/AWS/raw/v0.7/template.yaml .. note:: - This command will clone the repository and checkout the branch ``v0.7`` + Run this command on your local machine. This command will download the template.yaml file. 2. Navigate to the AWS console, and search for ``CloudFormation``. + .. note:: + The UI component deployed from this template is only supported in the following AWS Regions. Make sure that you create stack in the supported region. + Supported AWS regions: eu-north-1, ap-south-1, eu-west-3, us-east-2, eu-west-1, eu-central-1, sa-east-1, ap-east-1, us-east-1, ap-northeast-2, eu-west-2, ap-northeast-1, us-west-2, us-west-1, ap-southeast-1, ap-southeast-2, ca-central-1 + 3. Click on ``Stacks`` on the left hand side panel, and click on ``Create stack`` dropdown. Select ``With new resources (standard)`` to start creating a stack .. image:: resources/create_stack_start.png @@ -98,10 +132,10 @@ Deploy resources using the Cloud Formation template ATTACK Technique detector. Use version 0.4 Product ARN for the region in which CloudFormation stack is created. **ClusterModelARN:** The ARN of the subscribed model package for - Temporal Clustering. Use version 0.6 Product ARN for the region in which CloudFormation stack is created. + Temporal Clustering. Use version 0.7.1 Product ARN for the region in which CloudFormation stack is created. **FlowModelARN:** The ARN of the subscribed model package for MITRE - flow detector. Use version 0.6 Product ARN for the region in which CloudFormation stack is created. + flow detector. Use version 0.7 Product ARN for the region in which CloudFormation stack is created. **SuperuserEmail:** The email for admin user for UI @@ -109,12 +143,12 @@ Deploy resources using the Cloud Formation template **SuperuserPassword:** The password of the admin user for UI + **VRLLambdaImage:** The container image of the VRL Lambda that was pushed to ECR private repository in :ref:`setup_lambda_repository_single_command` + **WebContainerImage:** The container image of the subscribed marketplace UI product with tag ``market*``. The ``Web container image`` noted in the section :doc:`subscribe`. **NginxContainerImage:** The container image of the subscribed marketplace UI product with tag ``nginx-market*``. The ``Nginx container image`` noted in the section :doc:`subscribe`. - **VRLLambdaImage:** The container image of the VRL Lambda that was pushed to ECR private repository in :ref:`setup_lambda_repository` - The constraints for choosing the ``Cpu`` and ``Memory`` for the cluster can be found `here `__ Recommended value for parameter **ChunkSize** is below ``100000``. @@ -128,7 +162,11 @@ Deploy resources using the Cloud Formation template failure options``, select ``Roll back all stack resources`` for ``Behaviour on provisioning failure``. Select ``Delete all newly created resources`` for ``Delete newly created resources during a - rollback``. And then click on ``Next``. + rollback``. Expand the options for ``Stack creation options - optional`` and under ``Timeout``, enter ``15`` to set a max timeout of 15 minutes for the stack. And then click on ``Next``. + + .. image:: resources/stack_timeout.png + :alt: stack timeout + :align: center 8. Now in the ``Review and create`` page, you can review your parameters. At the bottom of the page, select all checkboxes for ``I @@ -138,10 +176,31 @@ Deploy resources using the Cloud Formation template 9. You can monitor the events of the cloud stack by clicking on the recently created cloud stack and going to the ``Events`` tab. + .. note:: + **Resource Creation Time:** The cloud stack will take approximately 10 minutes to complete the creation of all the resources. + 10. Once the cloud stack is completed successfully. You can start using - the products. + the products. Click on the ``Outputs`` tab for the recently created cloud + stack and note down the load balancer URL for the UI under ``LoadBalancerDNSName``. + Click on the link to open the UI. + + .. image:: resources/lb_url.png + :alt: lb url + :align: center Now all your resources are ready to be used. -You may now go to the step :doc:`end_to_end_test` to start testing -your application. + +Handling Multiple Inputs +------------------------- + +The pipeline will process files in the input folder sequentially in the order of upload. +Only one file will be processed at a time. Once a file is finished be processed the +pipeline will start with the next file in the queue automatically. + +.. note:: + **Small input files:** For best performance, it is not recommended to upload many + small files due to the startup time overhead of SageMaker jobs. + It is recommended to aggregate small inputs into larger input files. + + **Handling Large Input Files:** Currently the pipeline can handle upto 100,000 events in single input file. Be mindful of the input file that is used as input. diff --git a/docs/source/getting_started/prerequisites.rst b/docs/source/getting_started/prerequisites.rst index adfe4a0..2c8c899 100644 --- a/docs/source/getting_started/prerequisites.rst +++ b/docs/source/getting_started/prerequisites.rst @@ -13,7 +13,14 @@ Make sure that you have the required permissions for resources for the IAM user - ECR - IAM - CloudFormation +- Step Functions +To confirm you have the required permssion for the resources necessary to run the +pipeline you can check that with the following script. To run the script the iam user must have ``iam:SimulatePrincipalPolicy`` policy. + +.. code-block:: console + + $ wget -O- https://raw.githubusercontent.com/cypienta/AWS/v0.7/check_permissions.py | python Quotas ------ diff --git a/docs/source/getting_started/resources/lb_url.png b/docs/source/getting_started/resources/lb_url.png new file mode 100644 index 0000000..06ea6cc Binary files /dev/null and b/docs/source/getting_started/resources/lb_url.png differ diff --git a/docs/source/getting_started/resources/stack_timeout.png b/docs/source/getting_started/resources/stack_timeout.png new file mode 100644 index 0000000..9bbf81c Binary files /dev/null and b/docs/source/getting_started/resources/stack_timeout.png differ diff --git a/docs/source/getting_started/subscription.rst b/docs/source/getting_started/subscription.rst index a2757c9..982e1fa 100644 --- a/docs/source/getting_started/subscription.rst +++ b/docs/source/getting_started/subscription.rst @@ -42,7 +42,7 @@ Temporal Clustering :alt: Subscribe to technique detector :align: center -3. Click on ``Continue to configuration``. In the section ``Select your launch method``, select ``AWS CloudFormation``. Select the ``Software Version`` as ``0.6`` from the drop down. Select the ``Region`` in which you would want to deploy Cypienta products. Copy and make note of the ``Product Arn``. +3. Click on ``Continue to configuration``. In the section ``Select your launch method``, select ``AWS CloudFormation``. Select the ``Software Version`` as ``0.7.1`` from the drop down. Select the ``Region`` in which you would want to deploy Cypienta products. Copy and make note of the ``Product Arn``. .. image:: resources/model_arn_cluster.png :alt: Subscribe to flow detector @@ -66,7 +66,7 @@ MITRE ATTACK Flow Detector :alt: Subscribe to technique detector :align: center -3. Click on ``Continue to configuration``. In the section ``Select your launch method``, select ``AWS CloudFormation``. Select the ``Software Version`` as ``0.6`` from the drop down. Select the ``Region`` in which you would want to deploy Cypienta products. Copy and make note of the ``Product Arn``. +3. Click on ``Continue to configuration``. In the section ``Select your launch method``, select ``AWS CloudFormation``. Select the ``Software Version`` as ``0.7`` from the drop down. Select the ``Region`` in which you would want to deploy Cypienta products. Copy and make note of the ``Product Arn``. .. image:: resources/model_arn_flow.png :alt: Subscribe to technique detector @@ -90,7 +90,7 @@ Cypienta User Interface (UI) :alt: confirm subscribe :align: center -4. Select the ``Fulfillment option`` as ``ECS``. Select the ``Software version`` as ``v0.1.2``. Then click on ``Continue to Launch`` +4. Select the ``Fulfillment option`` as ``ECS``. Select the ``Software version`` as ``v0.2.2``. Then click on ``Continue to Launch`` .. image:: resources/to_launch.png :alt: to launch @@ -111,12 +111,12 @@ Cypienta User Interface (UI) --username AWS \ --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com - CONTAINER_IMAGES="709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:nginx-marketv0.0.3,709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:marketv0.1.2" + CONTAINER_IMAGES="709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:nginx-marketv0.0.3,709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:marketv0.2.2" for i in $(echo $CONTAINER_IMAGES | sed "s/,/ /g"); do docker pull $i; done Here the two images are: - - **Web container image:** 709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:marketv0.1.2 + - **Web container image:** 709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:marketv0.2.2 - **Nginx container image:** 709825985650.dkr.ecr.us-east-1.amazonaws.com/cypienta/cytech:nginx-marketv0.0.3 diff --git a/docs/source/getting_started/troubleshoot.rst b/docs/source/getting_started/troubleshoot.rst index dd760d3..88c3158 100644 --- a/docs/source/getting_started/troubleshoot.rst +++ b/docs/source/getting_started/troubleshoot.rst @@ -22,3 +22,94 @@ AWS console for the region you are using. Search for ``Service Quotas`` 3. Select the required instance type from the list and click on ``Request increase at account level``. + +How to delete stack +------------------- + +1. Navigate to the AWS console and search for ``CloudWatch``. Make sure you are in the same region in which you created CloudFormation stack. + +2. On the left hand side panel, under ``Logs``, click on ``Log groups``. Select all the check boxes for the ``Log groups`` that were created by the CloudFormation stack, click on ``Actions`` dropdown and click on ``Delete log group(s)``, and then click on ``Delete`` button. + +3. Next, search for ``S3`` in the AWS console search bar. + +4. Select the bucket that was created from the CloudFormation stack and click on ``Empty``. Type in ``permanently delete`` in the confirmation box and click on ``Empty``. + +5. Now search for ``CloudFormation`` in the the AWS console search bar. + +6. Open the stack that you want to delete and click on ``Delete``. Wait for the entire stack to be deleted before you move on to creating new stack. + + .. note:: + If there are any failures in deleting the stack, then ``Retry delete``. + + To speed up delete for stack, follow the optional steps below: + + 1. Navigate to AWS console and search for ``ECS`` and select ``Elastic Container Service``. + + 2. Click on the ECS cluster deployed from the stack. Select all the service from the ``Services`` tab and click on ``Delete service``. Check the box for ``Force delete`` and type in ``delete`` in the confirmation box and then click on ``Delete``. + + 3. Navigate to AWS console and search for ``EC2``. + + 4. Manually delete the running EC2 instance with name ``* - ``. Select all the pertinent instances, click on the ``Instance state`` dropdown and click on ``Terminate instance``. + +Common Mistakes +---------------- + +Some of the common errors that can result in the failure of the CloudFormation stack: + +- Duplicate S3 bucket name +- Incorrect arn for Models/UI +- Incorrect Image for VRL Lambda + +Duplicate S3 bucket name +~~~~~~~~~~~~~~~~~~~~~~~~ + +In the case of a duplicate S3 bucket name, delete the failed CloudFormation stack, +then choose a new globally unique S3 bucket name and recreate the stack. + +Incorrect arn for Models/UI +~~~~~~~~~~~~~~~~~~~~~~~~ + +In the case of an incorrect arn for models/UI, delete the failed CloudFormation stack, +then confirm the arns for all models and UI components as seen in :doc:`subscription` and recreate the stack. + +Incorrect Image for VRL Lambda +~~~~~~~~~~~~~~~~~~~~~~~~ + +In the case of an incorrect image for VRL lambda, delete the failed CloudFormation stack, +then ensure that you have the correct ECR Image URI and version number, and recreate the stack. + +Common errors +------------- + +CapacityError: Unable to provision requested ML compute capacity. Please retry using a different ML instance type. +~~~~~~~~~~~~~~~ + +If the SageMaker batch transform job fails for ``transform-job-cluster-*`` with the error +``CapacityError: Unable to provision requested ML compute capacity. Please retry using a different ML instance type.`` +the batch transform job can be retriggered manually. Follow the steps below to retrigger: + +1. Open the lambda function ``create_cluster``. + +2. Click on the ``Configuration`` tab, then click on ``Environment variables``. +Click on ``Edit`` button, and click on ``Add environment variable``. Under the ``Key`` text field enter ``batch_transform_job_suffix``, under ``Value`` text field enter any unique value. Limit the text value to length of 3. For example, ``1``. And, click on ``Save`` button. + +3. Open the S3 bucket created by the CloudFormation stack. Navigate to ``scratch/output/classification//``. + +4. Select the ``input.json``, click on ``Actions``, click on ``Copy``. On the Copy page, click on ``Browse S3``, click on ``Choose destination``, and then click on ``Copy``. + +5. This will trigger a new batch transform job. + +If the SageMaker batch transform job fails for ``transform-job-flow-*`` with the error +``CapacityError: Unable to provision requested ML compute capacity. Please retry using a different ML instance type.`` +the batch transform job can be retriggered manually. Follow the steps below to retrigger: + +1. Open the lambda function ``create_flow``. + +2. Click on the ``Configuration`` tab, then click on ``Environment variables``. +Click on ``Edit`` button, and click on ``Add environment variable``. Under the ``Key`` text field enter ``batch_transform_job_suffix``, under ``Value`` text field enter any unique value. Limit the text value to length of 3. For example, ``1``. And, click on ``Save`` button. + +3. Open the S3 bucket created by the CloudFormation stack. Navigate to ``scratch/output/cluster//``. + +4. Select the ``input_flow.json``, click on ``Actions``, click on ``Copy``. On the Copy page, click on ``Browse S3``, click on ``Choose destination``, and then click on ``Copy``. + +5. This will trigger a new batch transform job. diff --git a/docs/source/index.rst b/docs/source/index.rst index e1a48f6..457cafb 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -22,20 +22,31 @@ In this documentation, you will find detailed instructions for: .. toctree:: :maxdepth: 1 - :caption: Ex Integrations (SIEM, XDR, SOAR) + :caption: Cypienta UI + + deploy_ui/start_using + +.. toctree:: + :maxdepth: 1 + :caption: Splunk Integration splunk/splunk splunk/vrl splunk/output + +.. toctree:: + :maxdepth: 1 + :caption: Elastic Integration + elastic/elastic elastic/vrl elastic/output .. toctree:: :maxdepth: 1 - :caption: Cypienta UI + :caption: JIRA Integration - deploy_ui/start_using + jira/jira .. toctree:: :maxdepth: 1 diff --git a/docs/source/jira/jira.rst b/docs/source/jira/jira.rst new file mode 100644 index 0000000..a221f66 --- /dev/null +++ b/docs/source/jira/jira.rst @@ -0,0 +1,87 @@ +Configure JIRA +================= + +Prerequisites +------------- + +Make sure that you have deployed the Cypienta application detailed in :doc:`../getting_started/deploy` before integrating. + +Make sure that you note the JIRA API token, JIRA Username, JIRA URL, and JIRA Project key before setting up the lambda function. + +Setup JIRA connection +----------------------------------------------- + +Follow the steps below to add a lambda function that will create the jira issue. + +Create lambda function +~~~~~~~~~~~~~~~~~~~~~~ + +1. Navigate to the AWS console in the same region where your Cypienta application is deployed. Search for Lambda in the search bar and click on ``Create function`` + +2. Select ``Author from scratch``, write the ``Function name`` as ``create_jira`` with desired prefix and suffix. For ``Runtime`` select ``Python 3.11``, and select the ``Architecture`` as ``x86_64``. Expand the ``Change default execution role`` and select ``Use an existing role``. Click on ``Create function`` to create the function. + + .. image:: resources/create_function.png + :alt: create function + :align: center + +3. Scroll to the bottom of the lambda function that was just created and find the section ``Layers``. Click on ``Add a layer`` to add a layer. + + .. image:: resources/add_layer.png + :alt: add layer + :align: center + +4. Select ``AWS layers`` as ``Layer source``. Select ``AWSSDKPandas-Python311`` as ``AWS layers`` and select the version that is available in the dropdown. Click on ``Add`` to add the layer to the lambda function. + + .. image:: resources/layer_config.png + :alt: layer config + :align: center + +5. Now select the ``Configuration`` tab on the lambda function overview page and select ``General configuration`` from the left hand side panel. Click on ``Edit`` button to modify the values. + +6. Edit the ``Memory`` field with max value that is available. Edit the ``Ephemeral storage`` field with max value that is available. Edit the ``Timeout`` with max value that is available. And then, click on ``Save``. + + .. image:: resources/general_config.png + :alt: general config + :align: center + +7. Again on the ``Configuration`` tab on the lambda function overview page, select ``Triggers`` from the left hand side panel. Click on ``Add trigger`` button to add a trigger. + +8. For the trigger configuration, select the source as ``S3``. From the bucket dropdown, select the bucket that was created for the Cypienta application. Select ``All object create events`` for ``Event types``. Add Prefix as ``scratch/output/flow/`` and suffix as ``.json``. Check the box to acknowledge the message for recursive invocation and click on ``Add``. + + .. image:: resources/trigger.png + :alt: add trigger + :align: center + +9. Back on the ``Configuration`` tab, select ``Environment variables`` from the right hand side panel. Click on ``Edit`` to add variables. + +10. To add environment vairables, click on ``Add environment variable`` button and enter the following key, value. Finally, click on ``Save``. + + - Key: ``cluster_or_flow`` + Value: ``cluster`` + + - Key: ``event_threshold`` + Value: ``2`` + Description: The minimum number of events that must be present in a cluster for which a JIRA issue is to be created. + + - Key: ``jira_api_token`` + Value: ```` + + - Key: ``jira_lookup_object`` + Value: ``scratch/jira/issues.csv`` + + - Key: ``jira_project_key`` + Value: ```` + + - Key: ``jira_url`` + Value: ```` + Sample value: ``https://cypienta-demo.atlassian.net`` + + - Key: ``jira_username`` + Value: ```` + +11. Get the lambda function `create_jira `__ and copy paste the code in the ``Code`` tab in the ``Code source`` section. Click on the ``Deploy`` button to save the lambda function. + + +Now all the new clusters created will be pushed to the JIRA project that was configured in the environment vairables of the lambda function. + +Refer to :doc:`../splunk/splunk` for integrating JIRA with Splunk SOAR. diff --git a/docs/source/jira/resources/add_layer.png b/docs/source/jira/resources/add_layer.png new file mode 100644 index 0000000..c59326f Binary files /dev/null and b/docs/source/jira/resources/add_layer.png differ diff --git a/docs/source/jira/resources/create_function.png b/docs/source/jira/resources/create_function.png new file mode 100644 index 0000000..e0bbb13 Binary files /dev/null and b/docs/source/jira/resources/create_function.png differ diff --git a/docs/source/jira/resources/general_config.png b/docs/source/jira/resources/general_config.png new file mode 100644 index 0000000..7debd00 Binary files /dev/null and b/docs/source/jira/resources/general_config.png differ diff --git a/docs/source/jira/resources/layer_config.png b/docs/source/jira/resources/layer_config.png new file mode 100644 index 0000000..08fd67d Binary files /dev/null and b/docs/source/jira/resources/layer_config.png differ diff --git a/docs/source/jira/resources/trigger.png b/docs/source/jira/resources/trigger.png new file mode 100644 index 0000000..d8b5479 Binary files /dev/null and b/docs/source/jira/resources/trigger.png differ diff --git a/docs/source/lambda/functions.rst b/docs/source/lambda/functions.rst index 3043aaa..ffdf6a6 100644 --- a/docs/source/lambda/functions.rst +++ b/docs/source/lambda/functions.rst @@ -6,60 +6,75 @@ Functionality of Lambda Functions The fleet of lambda functions will be responsible for end-to-end flow for the Cypienta Sagemaker products. -1. **enrich_with_technique:** +1. **splunk_input:** + +- Get the input from splunk pushed to S3 path ``splunk_input/input`` and chunk it to tranform using VRL. +- Merge back transformed input after vrl_lambda processes all chunks and put it in the ``input/`` folder. + +2. **vrl_lambda:** + +- Transform data from CIM to CEF mapping. +- Save transformed alerts to S3. + +3. **skip_input:** + +- Get the input data from the ``input/`` S3 folder +- Check if an execution is running for step function. If execution is not running, start one. Else, add the current input to queue + +4. **enrich_with_technique:** - Get the input data from the ``input/`` S3 folder - Chunk the input, sanitize it in format as required for cluster model, encode node_features, encode other_attributes_dict, create mappings for internal ids to user given ids, mappings for chunk unique id to internal ids. - Enrich input with techniques. If the lookup table does not contain the specific technique, then start technique classification transform job per chunk -2. **process_enriched_with_technique:** +5. **process_enriched_with_technique:** - Get response from technique transform job and enrich the input with recognized techniques - Create input for the clustering model by adding node features if present. And save the resulting file to S3 -3. **update_lookup_table:** +6. **update_lookup_table:** - Update technique lookup table. -4. **create_cluster:** +7. **create_cluster:** - Read the input file saved to S3. If this is the first batch for the input file, then start clustering transform job. Else skip the file. -5. **process_cluster:** +8. **process_cluster:** - Read the response from clustering model. - Check if there is another batch that needs to run after the current response. If yes, then create input for the next batch, save to S3, and start clustering transform job. Else, extract agg_alert.json, cluster.json (for internal scratch) to S3, and create input for flow model and save to S3. -6. **create_flow:** +9. **create_flow:** - Triggered by input saved to s3 for flow model. Create flow transform job -7. **process_flow:** +10. **process_flow:** - Read response from the flow model. Save the flow_output.json to s3 (for internal scratch) - Clean up flow.json, cluster.json for user and save to ``output/`` folder. - Create enrich_alert_input.json and save to S3 (for internal scratch) -8. **create_campaign:** +11. **create_campaign:** - Read enrich_alert_input.json and create campaigns on UI -9. **create_jira:** +12. **save_feedback:** -- Read enrich_alert_input.json -- Read lookup for the JIRA issue to cluster id. -- If the cluster id already has JIRA created, and the status is ``open`` / ``in progress`` / ``to do``, overwrite the description with new details. If the status is not ``open`` / ``in progress`` / ``to do``, then create new JIRA issue with updated summary and description -- If the cluster id does not have JIRA created, then create JIRA issue with summary, description and attachment to subset of involved alerts +- Triggered by cut action performed on UI. +- Fetch involved events and campaigns from UI and update weights for node and event attributes. +- Create cluster ticket output for involved clusters, and save feedback. -10. **create_case:** +.. 13. **create_jira:** -- Read enrich_alert_input.json -- Read lookup for the Elastic case to cluster id. -- If the cluster id already has case created, and the status is ``open`` / ``in progress``, overwrite the description with new details. If the status is not ``open`` / ``in progress``, then create new case with updated summary and description -- If the cluster id does not have case created, then create case with summary, description. +.. - Read enrich_alert_input.json +.. - Read lookup for the JIRA issue to cluster id. +.. - If the cluster id already has JIRA created, and the status is ``open`` / ``in progress`` / ``to do``, overwrite the description with new details. If the status is not ``open`` / ``in progress`` / ``to do``, then create new JIRA issue with updated summary and description +.. - If the cluster id does not have JIRA created, then create JIRA issue with summary, description and attachment to subset of involved alerts -11. **save_feedback:** +.. 14. **create_case:** -- Triggered by cut action performed on UI. -- Fetch involved events and campaigns from UI and update weights for node and event attributes. -- Create cluster ticket output for involved clusters, and save feedback. +.. - Read enrich_alert_input.json +.. - Read lookup for the Elastic case to cluster id. +.. - If the cluster id already has case created, and the status is ``open`` / ``in progress``, overwrite the description with new details. If the status is not ``open`` / ``in progress``, then create new case with updated summary and description +.. - If the cluster id does not have case created, then create case with summary, description. diff --git a/docs/source/splunk/splunk.rst b/docs/source/splunk/splunk.rst index 72b646c..4a5b96a 100644 --- a/docs/source/splunk/splunk.rst +++ b/docs/source/splunk/splunk.rst @@ -1,6 +1,11 @@ Configure Splunk ================ +Prerequisites +------------- + +Make sure that you have deployed the Cypienta application detailed in :doc:`../getting_started/deploy` before integrating. + Getting AWS Access key ---------------------- @@ -85,6 +90,11 @@ To get search results of Splunk to AWS S3. Follow the steps below: :alt: Configure action for alert :align: center + + .. note:: + To make a quick test of the alert setup, select ``Run on Cron Schedule`` an type in the cron job schedule with reference to UTC-0 timezone. For example, if the current time is 13:46 UTC-0, then the ``Cron Expression`` can be set to 2 minutes past the current time. + ``48 * * * *``. This cron expression will run the alert at every 48th minute past the hour every hour in referene to UTC-0. If this is a one-off test, make sure to use dummy bucket and path to save the output, and make sure to disable the alert once test is successful. + 8. Add the ``Bucket name`` which was created using the CloudFormation template to save the results. For ``Object key``, enter ``splunk_input/input/%d-%b-%Y %H:%M:%S.json``. Select ``Account`` that you created on the configuration page from the dropdown. Finally click ``Save``. .. note:: diff --git a/docs/source/vrl/ai.rst b/docs/source/vrl/ai.rst index 84a9580..0fe3690 100644 --- a/docs/source/vrl/ai.rst +++ b/docs/source/vrl/ai.rst @@ -1,4 +1,4 @@ Using an AI model to generate mappings ---------------------------- +====================================== https://github.com/cypienta/data_mapper_model diff --git a/docs/source/vrl/transform.rst b/docs/source/vrl/transform.rst index fd99b0d..c63bdca 100644 --- a/docs/source/vrl/transform.rst +++ b/docs/source/vrl/transform.rst @@ -1,5 +1,5 @@ VRL mapping language & engine ---------------------------- +============================= Once you have the Elastic or Splunk logs stored in S3 in JSON Lines format, you can use the `VRL (Vector Remap @@ -8,7 +8,7 @@ your data to meet the format expected by the Cypienta end-to-end processing. VRL Transformations -~~~~~~~~~~~~~~~~~~~ +------------------- To apply a transformation to your log source using the VRL tool, you need to specify a VRL program file to transform your data as a string in @@ -16,7 +16,7 @@ the ``transforms`` key in your ``log_source.yml`` file. Write your VRL transformation script and save it as a ``.vrl`` file. Here, ``program.vrl`` Example: parsing JSON -^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~ To look at a simple example. Let's assume the following event. @@ -208,7 +208,7 @@ The resulting event: } Writing transformation VRL expressions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The input to your VRL expression is a single record from your data source. The output of the VRL expression is the transformed record. @@ -217,7 +217,7 @@ source. The output of the VRL expression is the transformed record. Using an AI model to generate mappings -~~~~~~~~~~~~~~~~~~~ +-------------------------------------- https://github.com/cypienta/data_mapper_model