Managing Integrations and Secrets in Dataloop 🔐

Welcome to your guide to managing integrations and secrets in Dataloop! Let's learn how to securely connect your external resources and manage sensitive information.

Understanding Integrations 🔗

Think of integrations as secure bridges between Dataloop and your external resources. They're your organization's VIP passes to safely access and manage external services! 🎫

What Can You Connect To? 🌐

Integrations allow your Dataloop organization to securely configure and access a variety of external resources:

Cloud Storage Services ☁️
- Google Cloud Platform (GCP)
- Amazon S3
- Microsoft Azure Blob
Container Registries 🐳
- AWS Elastic Container Registry (ECR)
- Google Container Registry (GCR)
- Google Artifact Registry (GAR)
Secure Token Service (STS) 🔑
And more exciting services! 🚀

Why Use Integrations? 🎯

When working with cloud services (like storage drivers), setting up an integration is your crucial first step. It's like creating a secure vault where you can:

Store access tokens safely 🔒
Manage service credentials efficiently 🗝️
Enable seamless connections between services 🤝

Setting Up Integrations 🛠️

Cloud Storage Integration ☁️

When working with cloud storage, setting up an integration is your first step. It's where you securely store your access credentials:

import dtlpy as dl

# Get your project
project = dl.projects.get(project_name='My-Project')

# Create the integration
project.integrations.create(
    integrations_type=dl.ExternalStorage.S3,  # Choose your cloud provider type
    name='my-cloud-integration',
    options={
        "key": "Access key ID",
        "secret": "Secret access key"
    }
)

Want to learn more about cloud storage? Check out our detailed guides:

Managing Secrets 🗝️

Understanding Secrets Manager

Our secrets manager is your vault for sensitive information. It's designed to:

Keep your credentials secure 🔒
Make them easily accessible in your code 🎯
Integrate smoothly with Pipelines and FaaS 🔄

Creating Key-Value Secrets ✨

Need to store a simple key-value pair? Here's how:

import dtlpy as dl

# Get your organization
organization = dl.organizations.get(organization_name='my-org')

# Create a key-value secret
organization.integrations.create(
    name='my-secret',
    integrations_type=dl.IntegrationType.KEY_VALUE,
    options={
        'key': "my_key",
        'value': "my_value"
    }
)

Using Secrets in Your Code 💻

When working with FaaS, you can use secrets to access your external resources. In the cloud environment, the secrets are saved as an environment variable.

import dtlpy as dl
import os

def my_function(item):
    # Access your secrets
    secret = os.environ.get('my-secret')
    # Your code here
    return item

For more details on using secrets in functions, check out our FaaS Security and Environment Chapter.

Updating Secrets 💻

You can update an existing Secret’s name, value, or provider using the SDK. This is particularly useful for automating credential rotation, updating expired keys, or adjusting secret metadata to improve service clarity.

import dtlpy as dl

integration = dl.integrations.get(integrations_id='', organization_id='')
integration.update(new_options={}, reload_services=True)

# Set reload_services=True to apply changes to all related services immediately.
# If reload_services is not provided, it defaults to False (services will not be reloaded automatically).

Best Practices 🌟

Naming Convention: Use clear, descriptive names for your integrations and secrets
Access Control: Only share secrets with those who need them
Regular Rotation: Update your secrets periodically for better security
Documentation: Keep track of what each secret is used for
Validation: Always validate your integrations after setting them up

Need More Help? 🤔

Check out our comprehensive documentation for more details on managing integrations and secrets.

Happy securing! 🚀