Managing Integrations and Secrets in Dataloop 🔐

Welcome to your guide to managing integrations and secrets in Dataloop! Let's learn how to securely connect your external resources and manage sensitive information.

Understanding Integrations 🔗

Think of integrations as secure bridges between Dataloop and your external resources. They enable your organization to safely connect to:

Cloud Storage (AWS S3, Azure Blob, GCP) ☁️
Container Registries (ECR/GCR) 🐳
Git Repositories (GitHub, GitLab, Bitbucket) 🎟️
And more!

Setting Up Integrations 🛠️

Cloud Storage Integration ☁️

When working with cloud storage, setting up an integration is your first step. It's where you securely store your access credentials:

import dtlpy as dl

# Get your project
project = dl.projects.get(project_name='My-Project')

# Create the integration
project.integrations.create(
    integrations_type=dl.ExternalStorage.S3,  # Choose your cloud provider type
    name='my-cloud-integration',
    options={
        "key": "Access key ID",
        "secret": "Secret access key"
    }
)

Want to learn more about cloud storage? Check out our detailed guides:

Managing Secrets 🗝️

Understanding Secrets Manager

Our secrets manager is your vault for sensitive information. It's designed to:

Keep your credentials secure 🔒
Make them easily accessible in your code 🎯
Integrate smoothly with Pipelines and FaaS 🔄

Creating Key-Value Secrets ✨

Need to store a simple key-value pair? Here's how:

import dtlpy as dl

# Get your organization
organization = dl.organizations.get(organization_name='my-org')

# Create a key-value secret
organization.integrations.create(
    name='my-secret',
    integrations_type=dl.IntegrationType.KEY_VALUE,
    options={
        'key': "my_key",
        'value': "my_value"
    }
)

Using Secrets in Your Code 💻

When working with FaaS, you can use secrets to access your external resources. In the cloud environment, the secrets are saved as an environment variable.

import dtlpy as dl
import os

def my_function(item):
    # Access your secrets
    secret = os.environ.get('my-secret')
    # Your code here
    return item

For more details on using secrets in functions, check out our FaaS Security and Environment Chapter.

Best Practices 🌟

Naming Convention: Use clear, descriptive names for your integrations and secrets
Access Control: Only share secrets with those who need them
Regular Rotation: Update your secrets periodically for better security
Documentation: Keep track of what each secret is used for
Validation: Always validate your integrations after setting them up

Need More Help? 🤔

Check out our comprehensive documentation for more details on managing integrations and secrets.

Happy securing! 🚀