Last updated

Managing Integrations and Secrets in Dataloop πŸ”

Welcome to your guide to managing integrations and secrets in Dataloop! Let's learn how to securely connect your external resources and manage sensitive information.

Understanding Integrations πŸ”—

Think of integrations as secure bridges between Dataloop and your external resources. They're your organization's VIP passes to safely access and manage external services! 🎫

What Can You Connect To? 🌐

Integrations allow your Dataloop organization to securely configure and access a variety of external resources:

  • Cloud Storage Services ☁️
    • Google Cloud Platform (GCP)
    • Amazon S3
    • Microsoft Azure Blob
  • Container Registries 🐳
    • AWS Elastic Container Registry (ECR)
    • Google Container Registry (GCR)
    • Google Artifact Registry (GAR)
  • Secure Token Service (STS) πŸ”‘
  • And more exciting services! πŸš€

Why Use Integrations? 🎯

When working with cloud services (like storage drivers), setting up an integration is your crucial first step. It's like creating a secure vault where you can:

  • Store access tokens safely πŸ”’
  • Manage service credentials efficiently πŸ—οΈ
  • Enable seamless connections between services 🀝

Setting Up Integrations πŸ› οΈ

Cloud Storage Integration ☁️

When working with cloud storage, setting up an integration is your first step. It's where you securely store your access credentials:

import dtlpy as dl

# Get your project
project = dl.projects.get(project_name='My-Project')

# Create the integration
project.integrations.create(
    integrations_type=dl.ExternalStorage.S3,  # Choose your cloud provider type
    name='my-cloud-integration',
    options={
        "key": "Access key ID",
        "secret": "Secret access key"
    }
)

Want to learn more about cloud storage? Check out our detailed guides:

Managing Secrets πŸ—οΈ

Understanding Secrets Manager

Our secrets manager is your vault for sensitive information. It's designed to:

  • Keep your credentials secure πŸ”’
  • Make them easily accessible in your code 🎯
  • Integrate smoothly with Pipelines and FaaS πŸ”„

Creating Key-Value Secrets ✨

Need to store a simple key-value pair? Here's how:

import dtlpy as dl

# Get your organization
organization = dl.organizations.get(organization_name='my-org')

# Create a key-value secret
organization.integrations.create(
    name='my-secret',
    integrations_type=dl.IntegrationType.KEY_VALUE,
    options={
        'key': "my_key",
        'value': "my_value"
    }
)

Using Secrets in Your Code πŸ’»

When working with FaaS, you can use secrets to access your external resources. In the cloud environment, the secrets are saved as an environment variable.

import dtlpy as dl
import os

def my_function(item):
    # Access your secrets
    secret = os.environ.get('my-secret')
    # Your code here
    return item

For more details on using secrets in functions, check out our FaaS Security and Environment Chapter.

Updating Secrets πŸ’»

You can update an existing Secret’s name, value, or provider using the SDK. This is particularly useful for automating credential rotation, updating expired keys, or adjusting secret metadata to improve service clarity.

import dtlpy as dl

integration = dl.integrations.get(integrations_id='', organization_id='')
integration.update(new_options={}, reload_services=True)

# Set reload_services=True to apply changes to all related services immediately.
# If reload_services is not provided, it defaults to False (services will not be reloaded automatically).

Best Practices 🌟

  1. Naming Convention: Use clear, descriptive names for your integrations and secrets
  2. Access Control: Only share secrets with those who need them
  3. Regular Rotation: Update your secrets periodically for better security
  4. Documentation: Keep track of what each secret is used for
  5. Validation: Always validate your integrations after setting them up

Need More Help? πŸ€”

Check out our comprehensive documentation for more details on managing integrations and secrets.

Happy securing! πŸš€