Meltano & Docker Compose

This file has been added to your project for convenience and reference only. Feel free to delete it.

Getting started

Start the services in the background:
```
docker compose up -d
```

Helpful commands

docker compose run meltano {subcommand}: Run a meltano CLI command inside your container.
docker compose logs: See all logs.
docker compose logs {service}: See logs for a particular service, e.g. meltano.

Optional services

If these services are not relevant to you, feel free to delete their commented sections.

Airflow

If you are using the Airflow orchestrator and would like to run it using Docker Compose, follow these steps:

Uncomment the airflow-webserver and airflow-scheduler services.
Start the new services:
```
docker compose up -d
```
Open the Airflow web interface at http://localhost:8080.

Production usage

A docker-compose.prod.yml file is included that represents a production-grade setup of a containerized Meltano project.

If this is not relevant to you, feel free to delete it.

Dependencies

The production configuration depends on a Dockerfile being present in your project.

If you haven't already, add the appropriate Dockerfile and .dockerignore files to your project by adding the docker file bundle:

meltano add files docker

Usage

Please ensure you do the following before deploying to production:

If you are using the Airflow orchestrator and would like to run it using Docker Compose, uncomment the Airflow services, network, and volume, and add psycopg2 to airflow's pip_url in meltano.yml as described in the "Deployment in Production" guide. If not, feel free to delete the commented sections.
Change the database password for meltano-system-db (and airflow-metadata-db): look for # CHANGE ME.
Update the database connection URIs under x-meltano-env (and x-airflow-env) to reflect the changed passwords.
Add any environment variables from .env and your local environment that are needed for production under x-meltano-env.
Change the image name and tag under x-meltano-image to something that makes sense for your project.

Start the services in the background:

docker compose -f docker-compose.prod.yml up -d

If you've made changes to your project and need to rebuild your project-specific image, run docker compose -f docker-compose.prod.yml up -d --build.

2.9 KiB Raw Blame History