Docker is Awesome!!!
I wish I knew Docker earlier, before going through the hassle of creating VMs for local ETL development and testing. Docker can make the whole setup even easier. It can be done in just a few commands, using a Vertica Dockerfile, created based on this. In addition to easy virtualization, Docker also enables the entire setup can be automated in a script, allowing it to be version-controlled (i.e., Infrastructure as Code).
Some notes about this Dockerfile, compared to
- Added new schema, new user and new role as examples. Avoid using
dbadminuser for development purpose.
- Added Java and Maven for Java-based ETL and automated test execution.
- Demonstrated running Bash and SQL scripts to initialize the container/database.
How to run
docker build, download Vertica Community Edition from https://my.vertica.com/ and place in the same folder as the
Dockerfile takes “vertica-7.2.3-0.x86_64.RHEL6.rpm” as the install file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
In Mac OSX, remember that the
entrypoint.sh file should have executable permission.
Otherwise, you might get the error “oci runtime error: exec: ”/entrypoint.sh": permission denied".
After changing the file permission, you have to rebuild the image with
docker build before
docker run again.
“Insufficient resources” error when running ETL
You might get “Insufficient resources to execute plan on pool general … Memory exceeded” error when running a large ETL script against the Vertica container.
For complex ETL, Vertica might need additional memory to execute the query plan.
Simply setting higher memory allocation using
--memory option of
docker run might NOT work if using Docker Toolbox.
To set higher memory allowance, stop the
docker-machine and set memory as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Note that after running the above commands,
docker-machine inspect still shows
To verify if memory is properly allocated as desired, run
free command, for example, inside the container to verify.