tactic docker version

[Diego]

Hi Gary,

I'am working on a better version too, suited for a docker compose script where tactic, apache and postgres are on different containers.
I've been a bit too busy at works and hadn't much time to work on it.
The biggest problem is that tactic uses psql directly for some operation, like the creation of a new database, another point is making it possible to delete and recreate a container while keeping all the data, this imply changing the tactic install script quite a bit or writing a new one just for docker.

In your case I think the tacticenv package is not correctly installed in the python path

There is also another problem related to the fact that the tactic install path was moved from /home/apache to /opt/tactic and the install script creates tactic_temp in /opt/tactic/tactic_temp but your dockerfile does:
COPY tactic_linux-conf.xml /opt/tactic/tactic_data/config/tactic-conf.xml
and your tactic_linux-conf.xml has:
<tmp_dir>/home/apache/tactic_temp</tmp_dir>

[remkonoteboom]

Hi Diego,

If you need some of these issues fixed/changed, please add corresponding issues to Github and I will gladly make sure it gets fixed in order to make this work

[Diego]

Hi Remko,
Just sent a small PR to make the install script use /opt/tactic for some folder that where still located in /home/apache. I will open an issue for the specific docker requirements tomorrow.

[gary]

Hi Gary,

I'am working on a better version too, suited for a docker compose script where tactic, apache and postgres are on different containers.
I've been a bit too busy at works and hadn't much time to work on it.
The biggest problem is that tactic uses psql directly for some operation, like the creation of a new database, another point is making it possible to delete and recreate a container while keeping all the data, this imply changing the tactic install script quite a bit or writing a new one just for docker.

In your case I think the tacticenv package is not correctly installed in the python path

There is also another problem related to the fact that the tactic install path was moved from /home/apache to /opt/tactic and the install script creates tactic_temp in /opt/tactic/tactic_temp but your dockerfile does:
COPY tactic_linux-conf.xml /opt/tactic/tactic_data/config/tactic-conf.xml
and your tactic_linux-conf.xml has:
<tmp_dir>/home/apache/tactic_temp</tmp_dir>

Thanks for your comments Diego. I've made those changes to my config, docker-compose and Dockerfile. Now I'll investigate my environment issue (python newbie.)

Regarding persistent data: Have a look at the Docker documentation for Volumes. It's what I'm doing with my set-up so data can remain separate from the running containers. No need to change the install script.

In the config I've changed localhost to db for the database host. I'm hoping the tactic container can pick this up for database operations -- but I haven't gotten far enough to confirm but I'm confident it can work.

---

tacticenv did get installed to the image/container...

Code:

docker-compose run -u root -v tactic:/opt/tactic --entrypoint bash tactic

root@b3fa94b64699:/opt/tactic# ls -lah /usr/local/lib/python3.7/site-packages/tacticenv/
total 20K
drwxr-sr-x 3 root staff 4.0K Aug  3 15:32 .
drwxr-sr-x 1 root staff 4.0K Aug  3 15:32 ..
-rw-r--r-- 1 root staff 2.0K Aug  3 15:32 __init__.py
drwxr-sr-x 2 root staff 4.0K Aug  3 15:32 __pycache__
-rw-r--r-- 1 root staff  437 Aug  3 15:32 tactic_paths.py

---

Noticed some pull requests were merged. Seems to get a little further...

Code:

tactic_1  | Traceback (most recent call last):
tactic_1  |   File "src/bin/startup_dev.py", line 24, in <module>
tactic_1  |     from pyasm.common import Environment, Config
tactic_1  |   File "/opt/tactic/tactic/src/pyasm/common/__init__.py", line 39, in <module>
tactic_1  |     from .js_wrapper import *
tactic_1  |   File "/opt/tactic/tactic/src/pyasm/common/js_wrapper.py", line 21, in <module>
tactic_1  |     from tactic_client_lib import TacticServerStub
tactic_1  | ModuleNotFoundError: No module named 'tactic_client_lib'

[Diego]

Just checked your git repo and I think the problem is in:
docker-compose.yml

you set two env variables:
            TACTIC_INSTALL_DIR: '/home/apache/tactic'
            TACTIC_DATA_DIR: '/home/apache/tactic_data'
if you check /usr/local/lib/python3.7/site-packages/tacticenv/__init__.py they are used to setup the sys.path
they should point to the new installation deault:
            TACTIC_INSTALL_DIR: '/opt/tactic/tactic'
            TACTIC_DATA_DIR: '/opt/tactic/tactic_data'

Regarding the Docker Volumes, yes they are very handy to keep the status when you re-deploy docker images and very useful when upgrading, but...

If you launch the tactic installation in the dockerfile it will try to create the sthpw db inside the same image and fail.
You should install tactic without the db creation (--install_db false) but then you will have to write some code in the docker entry point to check if the DB exists and cerate it if it doesn't we should add an option in the install.py to just create the sthpw db.
Still both tactic and the install.py script use os.system() to launch some postgres utilities like createdb and psql to create new dbs for projects (see src/pyasm/search/database_impl.py) which will fail if the database is running on a different machine or docker image.

We should try to solve these problems in tactic first.

I started my docker image mainly as a quick debug tool to test changes in the tactic code in a clean environment to avoid the "works on my machine" problem, this is why I never developed e better/proper image, but I guess we could join forces with Remko to add a  Dockerfile/docker-compose.yml directly into the tactic codebase, for them to become "official"

In my production environment I use a docker image for postgres but I prefer to run tactic in a python virtualenv.

[gary]

Just checked your git repo and I think the problem is in:
docker-compose.yml

you set two env variables:
            TACTIC_INSTALL_DIR: '/home/apache/tactic'
            TACTIC_DATA_DIR: '/home/apache/tactic_data'
if you check /usr/local/lib/python3.7/site-packages/tacticenv/__init__.py tey are used to setup the sys.path
the should be:
            TACTIC_INSTALL_DIR: '/opt/tactic/tactic'
            TACTIC_DATA_DIR: '/opt/tactic/tactic_data'

Sorry, I started working in a new branch -- but I also removed those env variables. Now I feel like there's progress:

Code:

...
tactic_1  |   File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
tactic_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
tactic_1  | psycopg2.OperationalError: FATAL:  database "sthpw" does not exist
tactic_1  |
docker-tactic_tactic_1 exited with code 1

Regarding the Docker Volumes, yes they are used to keep the status when you re-deploy docker images and they are very useful for upgrading an image, but...

If you launch the tactic installation in the dockerfile it will try to create the sthpw db inside the same image and fail.
You should install tactic without the db creation (--install_db false) but then you will have to write some code in the docker entry point to check if the DB exists and cerate it if it doesn't we should add an option in the install.py to just create the sthpw db.

This is what I'm doing to make the install work for the image. For database creation there's init scripts provided by the popular db images, such as postgres. We can use an sql file placed at ./postgres/data/ to create the sthpw db. This file is ingested during postgres container creation. I've used this feature on other projects with success (mysql/maraiadb.) Unfortunately I've not been able to locate init sql for Tactic.

Still bot tactic and the install.py script use os.system() to launch some postgres utilities like createdb and psql to create new dbs for projects (see src/pyasm/search/database_impl.py) which will fail if the database is running on a different machine or docker image.

One of the changes we could make is to have the database host be read from an environment variable, TACTIC_DBHOST for example. That way, in the docker-compose.yaml we can do: TACTIC_DBHOST: db
or whatever the container name is set to under services. I also need to add the postgres client to the tactic image.

We should try to solve these problems in tactic first.

I started my docker image mainly as a quick debug tool to test changes in the tactic code in a clean environment to avoid the "works on my machine" problem, this is why I never developed e better/proper image, but I guess we could join forces with Remko to add a  Dockerfile/docker-compose.yml directly into the tactic codebase, for them to become "official"

In my production environment I use a docker image for postgres but I prefer to run tactic in a python virtualenv.

That's what I like about the flexibility of Docker and the rapid iteration it can provide. I appreciate your comments! If you're looking at my messy code, look at the frompython branch: https://github.com/lyonritchie/docker-ta...frompython :-)

[gary]

A little further :-)

Code:

tactic@399433ac7df8:~/tactic/src/bin$ python startup_dev.py


Data Directory [/opt/tactic/tactic_data]
Asset Directory [/opt/tactic/assets]
Temp Directory [/opt/tactic/tactic_temp]
Config path [/opt/tactic/tactic_data/config/tactic-conf.xml]
Registering project ... admin
Registering project ... default
Initializing Workflow Engine
Starting Scheduler ....

Starting TACTIC v4.8.0.b04 ...

tactic@399433ac7df8:~/tactic/src/bin$

This, after editing tactic\src\src\install\template\config\tactic_linux-conf.xml so it contained the database hostname I wanted: the postgres container name and not localhost.

Then I was able to run the install.py script from within the tactic container (defaults) to initialize the database.

Not yet sure why startup_dev.py quietly dies though.

[Diego]

Hi Gary,
I can't find where you create and populate the sthpw in postgres. The current install.py uses the createdb command without passing the host and will fail if the DB server is not local. When you start tactic the sthpw database must exist.

I made few addition to the install.py script to make it possible to pass host and username, this make it possible to install tactic using a remote DB host or docker image as long as you have the postgres utilities locally installed, I will make a PR today or tomorrow after testing if everything works fine. The passed parameters are set in /opt/tactic/tactic_data/config/tactic-conf.xml as well.

The idea is that you just do the git checkout in the Dockerfile and launch the install.py at image deployment when the db is running, perhaps putting a check where if tactic is already installed because of persistent volumes we try an update and then launch the server. In this way we will be able to update a tactic deployment simply by stopping the images and doing a "docker pull ..."

[gary]

That sounds like a good strategy Diego. Looking forward to the updates.

I changed the build so it pulls in a custom config file where I set the database server value to "db" -- the postgres container name. Then, during install it is able to connect and run the db init functions. Seemed to work -- no errors reported.

[listy]

Anything new on related? Does it already can be used to install TACTIC 4.8 ?