Private Tools
Learn how to manage private tools in Trickest.
About Private Tools
Private Tools in Trickest allow for a tailored approach to security operations. Unlike public tools, private tools can be fully customized to meet specific automation needs and are not visible to the public within the Trickest Library.
This feature enhances cybersecurity operations teams to tailor their security workflows with private CLI (Command Line) tools.
This documentation includes a detailed step-by-step guide that will take you through every part of the process, from the initial import to the final integration into your workflows. Screenshots, code snippets, and detailed explanations will ensure you have all the necessary information.
Features and Benefits of the Private Tools
- Secured Environment: The Private tab ensures that the tools you import remain confidential and are not exposed to the public, safeguarding your custom solutions.
- Customization: It allows users to import and configure tools tailored to their unique operational needs, essential for specialized tasks that off-the-shelf software cannot address.
- Easy Management: Users can easily manage all of their private tools in one centralized location, simplifying the process of updating, configuring, and monitoring the usage of each tool.
- Integration:* Private tools can be integrated into custom workflows within the Trickest platform, providing the same automation and efficiency as public tools.
This feature is particularly effective for organizations that rely on proprietary or specialized software to conduct their operations. It offers them the flexibility to leverage the power of Trickest’s workflow automation while maintaining the integrity and confidentiality of their custom tools.
Prerequisites
Before proceeding, make sure you have:
- A Trickest account with access to private tool import enabled.
- Trickest Client installed
- The tool you wish to import, compliant with Trickest’s specifications
- Basic familiarity with Trickest workflows.
Accessing the Library
After logging into your Trickest account, locate and click on the Library button in the navigation menu.
You will see two primary tabs within the Library: Public
and Private
.
Viewing Your Private Tools
Once in the Private section, you will have a grid of all the imported tools. If this is your first visit or you have not yet imported any tools, this area may be empty.
Tool Details and Management
Click on the tool’s card to manage or get more information about a specific tool. This will take you to a screen to see the tool’s description, version, and configuration options.
Dockerfile and Trickest YAML
Each tool will typically consist of files organized in a specific structure to ensure compatibility with Trickest’s system.
Tool File Structure Overview
tool-name/
├── Dockerfile
├── tool.yaml
├── run.sh (optional)
Dockerfile
This is the file that will be used to generate the Docker image of the tool. If you are unfamiliar with the Docker and its concepts, you can learn more on this link.
To create efficient and light Docker images, follow best practices. But do keep in mind that although having small (in size) images (which means faster startup time) is essential, it is even more critical to have stable images. It’s all about finding a compromise here, but pick stability if you have to choose.
Downloading the tool
Use one of the methods listed in versioning conventions to download a specific version of the tool into the docker image.
Example Dockerfile
FROM golang:1.17.3-alpine AS build-env
RUN apk add --no-cache git
RUN go install -v github.com/author/tool-name@latest
FROM alpine:3.15.0
RUN apk -U upgrade --no-cache \
&& apk add --no-cache bind-tools ca-certificates
COPY /go/bin/tool-name /usr/local/bin/
ENTRYPOINT ["tool-name"]
run.sh
If the tool you use only supports stdin
as an input, check out Parameterizing stdin/stdout where you will learn how to create run.sh
and configure the tool to be compatible with Trickest platform.
ADD run.sh /app/
RUN chmod +x /app/run.sh
RUN apk add bash
ENTRYPOINT ["/app/run.sh"]
Input/Output folders
You need to create two folders: /hive/in
and /hive/out
in the final image. These will be used by the platform to store and manage the tool’s input/output.
Tips for better Docker images
- Compiled languages (e.g. golang)
Use multi-stage builds to reduce the final image size Use -alpine images when possible
-
Python
-
You should almost always use
python:<version>-slim.
This is a variant that has less packages installed, resulting in a smaller image. -
If you run into any problems with -slim, the official image should be your second choice.
-
Try to stay away from
alpine
, because it might cause some unexpected issues. -
Tagging
Remeber to always tag your images properly according to the versioning guide.
Trickest Tool YAML
trickest.yaml Configuration File
The trickest.yaml
file is essential for integrating tools into the Trickest platform, specifically designed for the Workflow Editor. This YAML file contains all the necessary data for the tool to be displayed appropriately and configured within the platform.
Overview of trickest.yaml Structure
The YAML file is structured to provide comprehensive details about the tool, including its functionality, usage, and output handling. Below is a detailed explanation of each field in the trickest.yaml file:
Example trickest.yaml
name: my-private-tool
description: This is an example of a private tool
category: Recon
source_url: https://github.com/my-tool
docker_image: user/image:tag
command:
output_parameter: -o
output_type: file
license_info:
name: MIT
url: https://github.com/example-repository/LICENSE.md
inputs:
input-parameter:
command: -d
description: This is description of input parameter
order: 0
visible: true
type: string
Tool Identification and Description
Name | Description | Required | Example |
---|---|---|---|
name | Name of the tool | Yes | amass |
description | Short description of the tool. The “About” section of GitHub is an excellent starting point if available | Yes | The OWASP Amass Project performs network mapping of attack surfaces and external asset discovery using open source information gathering and active reconnaissance techniques. |
category | High-level category of the tool | No | Recon |
source_url | Original repository’s URL | Yes | https://github.com/OWASP/Amass |
Docker Image and Command Execution
Name | Description | Required | Example |
---|---|---|---|
docker_image | Docker image URL. We use quay.io for hosting images | Yes | quay.io/trickest/amass:v3.10.5 |
command | Command that should be executed on the container when the node runs | Yes | /bin/amass enum |
output_parameter | Command line parameter that designates the tool’s output path | Yes | -o |
output_type | Output type (file or folder) | Yes | file |
license_info.name | Name of the tool’s license | Yes | Apache 2.0 |
license_info.url | URL of the tool’s license | Yes | https://github.com/OWASP/Amass/blob/main/LICENSE |
:latest.
Parameters
The parameters structure is as follows:
Name | Description | Example |
---|---|---|
command | Command line parameter used by the tool | -d |
name | Name to identify the parameter in the node | domain |
parameter_type | Type of the parameter (string, file, folder, or boolean) | string |
description | Short description of the parameter | Domain names separated by commas |
order | The index of the parameter. The order is followed when building the final command | 0 |
Versioning Conventions
Part of our quest for keeping workflows stable is proper versioning of the tools. It’s important to provide users with the latest and greatest updates to their tools by default but also allow them to revert to a previous version if need be.
You must always tag your image before pushing it in at least one of the following ways:
Git Commit Short Hash
This is the recommended way that should be used for all tools as it’s a unique value and you can easily use it to match a container image to its source. If a tool is hosted on a public git repository, it should use this convention by default, preferably in addition to the others listed below.
docker tag <image_id> <image_name>:<short_commit_hash>
GitHub releases
If the tool has an official release cycle, you should follow the same versioning convention it uses. You can use wget to download the release into the image using the following command
wget https://github.com/author/tool/releases/download/<version>/tool.zip
apt-get
If a tool is available on an apt repository, use this command to install a specific version
apt-get install tool=<version>
go install
For Go tools hosted on git repositories, use go install to install a specific version or a specific commit
go install github.com/author/tool@<version OR commit hash>
pip
For Python tools hosted on pypi.org, use pip to install a specific version
pip install tool==<version>
gem
For Ruby tools hosted on rubygems.org, use gem to install a specific version
gem install tool -v <version>
npm
For JavaScript tools hosted on npmjs.com, use npm to install a specific version
npm install -g tool@<version>
Old Versioning
End users can access older versions of the tools through the git history of the tool’s corresponding trickest.yaml file.
Notes
- If a tool has more than one active branch (e.g. main and dev), each branch should be treated as a separate tool.
- If a tool has more than one active fork, each one should be treated as a separate tool.
- No new tags should be created unless the code on the original repo (referenced in source_url in trickest.yaml) changes. Updates anywhere else should keep the same tag.
Parameterizing STDIN/STDOUT
The platform currently needs explicit command line parameters for input and output:
tool -i input.txt -o output.txt
rather than using STDIN/STDOUT
:
cat input.txt | tool > output.txt
Suppose your tool expects input/output in the latter format and does not have an explicit parameter for input/output. In that case, you can write a shell one-liner to emulate the existence of a parameter in your Docker image, like this one:
cat "${@: 1:1}" | tool "${@: 2:$#-2}" | tee "${@: $#:1}"
Explanation:
@
is a list of all the input parameters passed to the script.${@: 1:1}
follows the format${@: START:COUNT}
, which means “starting from the parameter atSTART
(index 1 here), getCOUNT
(1 in this case) parameters,” which is practically equivalent to$1
, the first parameter.- The first command
cat "${@: 1:1}"
will output the content of the file passed in the first parameter. $#
refers to the number of parameters passed to the script.tool "${@: 2:$#-2}"
passes all the parameters totool
except the first and last one.tee "${@: $#:1}"
passes the last command totee
.
So when it all comes together, this script allows us to use the following command:
./run.sh input.txt -p1 v1 -p2 v2 output.txt
and it will be equivalent to:
cat input.txt | tool -p1 v1 -p2 v2 | tee output.txt
Templates
STDIN only
If the tool expects input parameter through STDIN
but accepts an output parameter, i.e., the original command follows this format:
cat input.txt | tool -o output.txt
Use this:
cat "${@: 1:1}" | tool "${@: 1:$#-1}"
STDOUT only
If the tool accepts an input parameter but writes the output to STDOUT
, i.e., the original command follows this format:
tool -i input.txt | tee output.txt
Use this:
tool "${@: 1:$#-1}" | tee "${@: $#:1}"
Both STDIN and STDOUT
If the tool expects an input parameter through STDIN
and writes the output to STDOUT
, i.e., the original command follows this format:
cat input.txt | tool | tee output.txt
Use this:
cat "${@: 1:1}" | tool "${@: 2:$#-2}" | tee "${@: $#:1}"
Where to Save This Script
Save the script to a separate file named run.sh
and add it to the module’s folder (don’t forget to add the shebang #!/bin/bash
).
#!/bin/bash
cat "${@: 1:1}" | tool "${@: 2:$#-2}" | tee "${@: $#:1}"
Changes in trickest.yaml
Main command
Use the script in the command
parameter.
Parameters
The input and output parameters must be adjusted as follows:
- The
output_parameter
should be set to>
. - The
output_type
should be set tofile
. - The input parameter’s command should be left empty.
command: 'cat "${@: 1:1}" | tool "${@: 2:$#-2}" | tee "${@: $#:1}"'
output_parameter: '>'
# This is important
output_type: file
inputs:
- name-of-parameter:
command:
type: file
description: DESCRIPTION
order: 0
visible: true
Dockerfile
The run.sh
script should be copied to the docker image on build and used as an ENTRYPOINT
.
ADD run.sh /app/
RUN chmod +x /app/run.sh
RUN apk add bash
ENTRYPOINT ["/app/run.sh"]
Example Tool Configuration and Import
trickest.yaml
The trickest.yaml
file is a structured configuration that defines how a tool will be integrated and function within the Trickest platform.
This file includes essential details like the tool’s name, description, category, and technical specifics such as the Docker image URL and command execution parameters. It serves as a blueprint for the Trickest system to recognize and properly handle the tool, ensuring it operates seamlessly within your workflows.
name: tool-name
description: Tool short description
category: Recon
source_url: https://github.com/author/tool-name
docker_image: quay.io/trickest/tool-name:latest
command: "/bin/tool-name"
output_parameter: "-o"
output_type: file
inputs:
parameter-1:
command: -d
description: string-input for tool
order: 0
visible: true
type: string
parameter-2:
command: -file
description: file-input for tool
order: 0
visible: true
type: string
license_info:
name: MIT
url: https://github.com/author/tool-name/blob/1234567890123456789/LICENSE
Dockerfile
For Trickest private tools, a Dockerfile
is essential for defining the environment in which your tool will run. It specifies the base image, necessary dependencies, build instructions, and how the tool should be executed. The Dockerfile ensures that your tool can be consistently deployed and run within the Trickest platform, regardless of the underlying infrastructure.
FROM golang:1.17.3-alpine AS build-env
RUN apk add --no-cache git
RUN go install -v github.com/author/tool-name@latest
FROM alpine:3.15.0
RUN apk -U upgrade --no-cache \
&& apk add --no-cache bind-tools ca-certificates
COPY --from=build-env /go/bin/tool-name /usr/local/bin/
ENTRYPOINT ["tool-name"]
Once your Dockerfile is ready, the following steps involve building and pushing the Docker image to a registry. This process is critical for making the tool available for integration into Trickest workflows.
Here’s how you can do it:
Log in to Docker Registry
Use the docker login
command to authenticate with the Docker registry where you intend to push your image.
docker login quay.io # or any other registry
Enter your username and password when prompted.
Build the Docker Image
Navigate to the directory containing your Dockerfile and build the image using the docker build
command:
docker build -t quay.io/username/tool-name:tag .
Replace username
, tool-name
, and tag
with your Docker registry username, your tool’s name, and the image’s version or tag.
Push the Image to the Registry
After successfully building the image, push it to the Docker registry using docker push
:
docker push quay.io/username/tool-name:tag
Ensure the tag used matches the one specified in the build step.
By completing these steps, your custom tool image will be available in your Docker registry, ready to be integrated into Trickest as a private tool. This process is essential for maintaining version control and ensuring consistent deployment of your tools across various environments.
Adding a Private Tool Through trickest-cli
Adding a new private tool to the Trickest platform using the trickest-cli is straightforward. This method allows you to integrate custom tools into your Trickest workflows efficiently.
Steps to Add a New Tool
Navigate to the Directory
Use the cd
command to navigate to the directory where your trickest.yaml
file is located.
Run the Create Command
TRICKEST_TOKEN
to Trickest CLI or have it set as an environment variable. You can find your token at this pageExecute the following command to create a new private tool integration in Trickest:
trickest tools create --file tool.yaml
tool.yaml
with the path to your YAML file.Confirmation and Testing
After running the command, you should receive a confirmation that your tool has been successfully created.
It’s a good practice to test the tool in a workflow to ensure it has been configured correctly.
Managing Your Tool
Once your tool is added, you can manage it using various commands in the Trickest-CLI
:
-
Update Tool: To update the tool, modify your
trickest.yaml
file and then run:trickest tools update --file tool.yaml
-
List Tools: To list all your private tools, use:
trickest tools list
Add
--json
to display the output in JSON format. -
Delete Tool: If you need to delete a tool, use:
trickest tools delete --name tool-name
By following these steps, you can efficiently manage the lifecycle of your private tools within the Trickest platform, enhancing the capabilities and customizability of your workflows.