Some difficulty installing v4.7.0.b02 in Ubuntu Server 18.04 LTS AWS EC2 instance

[FordSubmariner]

Hi all, and thanks for making and sharing TACTIC under a free and open source software license.

I hope everyone here is able to stay healthy and safe in the pandemic.

I'm brand new to TACTIC, am in the middle of trying to install it in a Ubuntu Server 18.04 LTS (HVM) AWS EC2 instance.

I found http://community.southpawtech.com/docs/s...c-general/ and http://forum.southpawtech.com/showthread.php?tid=90 and I've implemented almost all of the instructions I see at one or the other URL, but I have some basic questions about installing TACTIC in Ubuntu (where Apache2 is set up with a "sites-available" and a "sites-enabled" (et. al.) directory tree under /etc/apache2. So the TACTIC instruction step reading:

"*** Installation of TACTIC completed at [/home/apache] ***

Next, please install the Apache Web Server and then copy the Apache config extension
[/home/apache/tactic_data/config/tactic.conf] to the Apache web server config area. e.g.
/etc/httpd/conf.d/"

is not strictly applicable to Ubuntu. There is no /etc/httpd/conf.d/ directory. And it's not just that "httpd" has been replaced by "apache2" (I think I could handle that on my own), but my trouble comes from comparing the default apache2 website ("It Works!") and its config file (which includes Apache Directives like "DocumentRoot") whereas .../TACTIC-4.7/src/install/apache/tactic.conf contains no such Directive.

I guess part of my question is about things like Apache Virtual Hosts and Directives like "DocumentRoot" and "ServerName" and why these Directives are absent from tactic.conf (perhaps because all apache is doing is referring to TACTIC?).

When I installed TACTIC (by running "sudo python3 install.py" from TACTIC-4.7/src/install/) I chose all defaults (including /home/apache as install directory), I noticed that tactic.conf seems to assume an install directory of "/opt/tactic/" although the install script did put a bunch of files in /home/apache (though not an "assets" directory like I see at /opt/tactic/assets). So it looks like I may? need to make some other changes to tactic.conf, then copy tactic.conf to an appropriate location such that an apache2 Virtual Host can be added (with a2ensite) to the apache2 configuration, then reload the apache configuration. But it is not obvious to me exactly how to go about doing those steps in this Ubuntu 18.04 instance.

My latest (as of a few minutes ago) thinking is that TACTIC needs no VirtualHost Directive or config files for Apache2. Based on that, I just added tactic.conf to /etc/apache2/conf-available and then did sudo a2enconf tactic followed by systemctl reload apache2. (I don't know how to format text as code in this forum software; does it support markdown?)

After that I did:
ubuntu@...$ sudo su -s /bin/bash www-data
www-data@...$ python3 /home/apache/tactic/src/bin/startup_dev.py
Data Directory [/home/apache/tactic_data]
Asset Directory [/opt/tactic/assets]
Temp Directory [/opt/tactic/tactic_temp]
Config path [/home/apache/tactic_data/config/tactic-conf.xml]
Registering project ... admin
Registering project ... default
Initializing Workflow Engine
Starting Scheduler ....

Starting TACTIC v4.7.0.b02 ...

So I guess I have TACTIC_dev running. And I'm able to connect to my web server from a remote client using Firefox, and I was prompted (apparently by TACTIC_dev but there were broken images on this login page) to enter username and password (admin/tactic and verify password tactic), but after logging in that way, all I see is "ADMIN >>" in the upper left of the web page (everything else is blank/white). If it helps, I'll send anyone who is interested the URL of my TACTIC server in a PM so you can see for yourself. I think that a screenshot would contain no additional information than what I've posted in this paragraph.

I've found that instructions at http://community.southpawtech.com/docs/s...c-general/ which read:

"The file tactic_paths.py will be created in the following directories:

Linux

/usr/lib/python#.#/site-packages/tacticenv"

are incorrect for me. My Ubuntu 18.04 instance has python3.6 installed, yet neither /usr/lib/python3.6/ nor /usr/local/lib/python3.6/ (where python supporting modules like Pillow, lxml, pycryptodomex, requests, pytz, and psycopg2 were installed when I used pip3 to install them) include tactic_paths.py and I wonder if this could be a part of the reason for the broken images and blank page I'm seeing.

Thanks again for making and sharing TACTIC. It looks like a wonderful tool! I hope to have a fully functional TACTIC server running soon, and I look forward to experimenting with TACTIC!

Kind Regards,
Kevin Ford

[Celton McGrath]

Hey FordSubmariner,

For your error:

"but after logging in that way, all I see is "ADMIN >>" in the upper left of the web page (everything else is blank/white)."

My best guess is that there is a 403 error and Apache is blocking access to Javascript files required to run the front-end of TACTIC.

I think this because this is a common error, and as you described, Apache for TACTIC doesn't seem fully configured.

If you open the Javascript console for your TACTIC web page, see if there are any 403 errors.

Let me know if you have any console errors, and I can give you more info on how to configure Apache from there.

Best,

Celton

[FordSubmariner]

Hi Celton, and thanks for your quick reply.

I'm unfamiliar with the Javascript console, but I think you may mean a panel I get Firefox 75 to show me when I click Tools->Web Developer->Web Console.

In that log console for my TACTIC page, I see several errors like:

"Loading failed for the <script> with source “http://x.x.x/context/spt_js/applet.js?ver=4.7.0.b02”."
...
"Loading failed for the <script> with source “http://x.x.x/context/spt_js/chart/utils.js?ver=4.7.0.b02”."
ReferenceError: spt is not defined
ReferenceError: spt is not defined


But I see no 403 errors here (unless perhaps I just don't know what a 403 error is, but I filtered the output for "403" and found nothing resulting).

In Firefox, I also did Tools->Web Developer->Browser Console.

In that log console for my TACTIC page, I see several errors like:
"PushDB: update: Ignoring invalid update 0f73a593-7ac2-6646-930f-0315526b9f0c null PushDB.jsm:418
PushService: receivedPushMessage: Error notifying app AbortError PushService.jsm:829
...
Content Security Policy: Ignoring “http:” within script-src: ‘strict-dynamic’ specified
OpenGL compositor Initialized Succesfully. Version: 2.1 INTEL-10.32.48 Vendor: Intel Inc. Renderer: Intel Iris Pro OpenGL Engine FBO Texture Target: TEXTURE_2D 2
NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIInterfaceRequestor.getInterface] 2 network-response-listener.js:84
Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified"


There are many others too (19 errors, 42 warnings, some log entries, et. al.) I also filtered this console for "403" and found nothing resulting.

Does this help you figure out what I'm doing wrong? Thanks again for your suggestions. Although I still have the same problem, your suggestions are certainly teaching me a lot about browser features I was previously unaware of, and I very much appreciate that. And I'm optimistic that with a bit more troubleshooting and help from this forum, I'll soon be able to solve the problem.

I did set the devtools.chrome.enabled preference to true in about:config (it was previously false). I'm not sure if that helped or what.

[Celton McGrath]

Hi FordSubmariner,

Sorry, the console is a common tool for web developers.

Whether or not it is a 403, it appears that the Javascript files are not being served by Apache. You can confirm this by going trying to access a URL like,
http://x.x.x/context/spt_js/applet.js?ver=4.7.0.b02

and see if it displays the raw Javascript file.

I need to test on a Ubuntu machine, but I assume that the configuration will be something like placing the httpd tactic.conf file in /etc/apache2/sites-available/
The apache user also needs access to be able to read from the directories and files defined in this file:
/assets, /context, and /plugins.

[remkonoteboom]

"I guess part of my question is about things like Apache Virtual Hosts and Directives like "DocumentRoot" and "ServerName" and why these Directives are absent from tactic.conf (perhaps because all apache is doing is referring to TACTIC?)."

The tactic.conf file works in addition to, not in place of the httpd.conf file for apache.  The folder /etc/httpd/conf.d folder for RedHat style distributions is the location that is used for modules.  This path may vary depending on the distribution and even version of distribution.  Ubuntu as I recall from a while ago is different, but there is a standard location.  This link:

https://www.digitalocean.com/community/t...debian-vps

seems to indicate that it is /etc/apache2/conf.d.  At any rate the tactic.conf file does not need a DocumentRoot because apache communicates with TACTIC through a reverse proxy through the load balancer.  These are the lines that handle this:

Code:

# Using the ProxyPass directives
<Proxy balancer://tactic>
BalancerMember http://localhost:8081/tactic
BalancerMember http://localhost:8082/tactic
BalancerMember http://localhost:8083/tactic
</Proxy>
ProxyPass /assets !
ProxyPass /context !
Proxypass /doc !

ProxyPass /tactic balancer://tactic/
ProxyPass /projects balancer://tactic/

ProxyPass / balancer://tactic/

The last line is what redirects "/" which is the equivalent to DocumentRoot and sends it to the balancer.

[remkonoteboom]

The tactic_paths.py file is located somewhere in the site-packages folder for your Python3 version.  This will depend greatly on version (python2 vs python3) and which distribution you are using (Ubuntu, etc).  To find out if it is even installed, you can run python from the python command line

Code:

Python 3.7.7 (default, Mar 13 2020, 10:23:39)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tacticenv
>>> tacticenv.__path__
['/usr/lib/python3.7/site-packages/tacticenv']
>>>

This shows the path of this file.  If it wasn't installed correctly, you would not even see this module and you would not even be able to run "python startup.py" as it would error our immediately not being able to find the "tacticenv" module.

Your problem is likely because the apache cannot see the "context" folder.

"Loading failed for the <script> with source “http://x.x.x/context/spt_js/chart/utils.js?ver=4.7.0.b02”."

If you click on this link to open it (http://x.x.x/context/spt_js/chart/utils....=4.7.0.b02), it will tell you more what the problem is.  Alternatively, you can go to the error_log files for apache (for RHEL, these are located in /var/log/httpd/).  My guess is that this is a permission error when the user running TACTIC is not permitted to see the "context" folder in the TACTIC source.  The following lines give this permission to apache:

Code:

<Directory "/opt/tactic/tactic" >
    Options FollowSymLinks
    AllowOverride None
    # Apache < 2.4
    #Order Allow,Deny
    #Allow from All
    # Apache 2.4
    Require all granted
</Directory>

However, it could also be a file permission.  Make sure the user running apache is allowed to read the TACTIC source folder "context".

[FordSubmariner]

Thanks again Celton, and thank you remkonoteboom.

This has all been very helpful to me.

I now think that I was mistaken when I earlier posted that tactic_paths.py was not where it should be. Yesterday I think I forgot to run updatedb after installing TACTIC and so was looking at the output from locate which had been based on a minutes-earlier run of updatedb because locate now (after updatedb ran it's cron daily run last night I guess) shows me the location that tactic_paths.py was installed: /usr/lib/python3/dist-packages/tacticenv/tactic_paths.py

remkonoteboom, your suggestion to import tacticenv in a python3 console helped me find it, so thank you for that.

In this Ubuntu instance, the user running apache is "www-data".

The TACTIC install.py seems to have defaulted to an install location of /home/apache (I don't think that's where most apache2 document trees are located) and that's ok with me. The TACTIC install.py seems to have figured out that "www-data" is the user running apache in this distro, and all directories and files under /home/apache are owned by "www-data".

When I earlier mentioned the browser console log having a problem loading the script "http://x.x.x/context/spt_js/chart/utils.js?ver=4.7.0.b02", I now think (thanks again to remkonoteboom's guidance) that this script must be /home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart/utils.js and every file and directory in the path I list here is owned by "www-data":

-rw-rw-r-- 1 www-data root 3318 Dec  5 18:47 /home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart/utils.js
drwxrwxr-x 2 www-data root 4096 Dec  5 18:47 /home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart
drwxrwxr-x 11 www-data root 4096 Dec  5 18:47 /home/apache/tactic_src_4.7.0.b02/src/context
drwxrwxr-x 16 www-data root 4096 Dec  5 18:47 /home/apache/tactic_src_4.7.0.b02/src
drwxrwxr-x 5 www-data root 4096 Dec  5 18:47 /home/apache/tactic_src_4.7.0.b02

And I'm launching startup_dev.py as that same "www-data" user. But I seem to recall reading yesterday that there is a separate "TACTIC" user? Or am I recalling incorrectly on this?

Celton and remkonoteboom, I tried your suggestion of trying to access a URL like "http://x.x.x/context/spt_js/chart/utils.js?ver=4.7.0.b02" from the browser console, and as you predicted, that failed. When I tried, I saw a "You don't have permission to access this resource." and now I see that the title of that new (error) page is "403 Forbidden". Celton, I guess that's what you meant by 403? Sorry if I was slow to pick up on that.

But I don't understand why it's forbidden because I'm running startup_dev.py as "www-data" and /home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart/utils.js is owned by "www-data" and with file metadata listing "rw" permissions for the file owner. I'm thinking it must be an apache configuration problem?

remkonoteboom, your point that "The following lines give this permission to apache:" and your listing the lines from tactic.conf was helpful to me because I looked in /opt/tactic/tactic for context/spt_js/chart/utils.js and it's not there. So based on that, I changed:

<Directory "/opt/tactic/tactic" >

to

<Directory "/home/apache/tactic" >

because that's where I find /home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart/utils.js (/home/apache/tactic is a SymLink pointing to /home/apache/tactic_src_4.7.0.b02 )

I then reloaded my apache configuration:
sudo systemctl reload apache2

and tried everything again. I still see the same TACTIC UI I reported before, and the 403 errors are still there when I attempt to access them from the browser console. At first I was thinking that perhaps the browser had cached those 403 error pages, so I used <shift>-reload-page to try and bypass any browser cache, but still, I get the same 403 error pages.

I then even tried restarting my apache2 server:
sudo systemctl restart apache2

And it's still doing the same thing.

There is also the section of tactic.conf:
<Directory "/opt/tactic/assets" >
    Options FollowSymLinks
    AllowOverride None
    # Apache < 2.4
    #Order Allow,Deny
    #Allow from All
    # Apache 2.4
    Require all granted
</Directory>


And this directory "/opt/tactic/assets" was (before I changed it moments ago) initially owned by root:
$ ls -l /opt/tactic
total 8
drwxr-xr-x 2 root    root 4096 May 17 18:44 assets


But when I found that, I changed its owner to www-data (even though it's empty) just in case, and reloaded my apache configuration. It's still doing the same thing.

Any other ideas? I really appreciate all the help and I'm still keen to get this up and running and confident that I'll find the problem soon.

Many thanks again for all the suggestions!

[remkonoteboom]

Welcome to the world of Apache permissions! Apache tends to be very restrictive for security reasons: any issue anywhere will cause it to reject the request.

So the basic known problem is that this link:

http://x.x.x/context/spt_js/chart/utils....=4.7.0.b02

is not permitted. Until this is solved, nothing else matters. In your case, this file is located in:

/home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart/utils.js

(I agree that this really should be stored here. It is vestigial from a long time ago. Now we generally install in either /home/tactic or more recently /opt/tactic)

Your user, www-data, is likely not permitted to see this path as seen through apache. This is usually due to file permissions or httpd config setup. Since you have symbolic links, all your paths in <Directory> need to have "Options FollowSymLinks"

The other line to check is in the tactic.conf file:

Alias /context /home/apache/tactic/src/context

This maps /context url to this folder (the default tactic.conf will have this set to: /opt/tactic/tactic/src/context). You will need to make sure that all of the references to "/opt/tactic/tactic" are switched to "/home/apache/tactic".

Finally, you should check the httpd error logs. They are sometimes helpful, but likely it will just that the path "/home/apache/tactic_src_4.7.0.b02/src/context/spt_js/chart/utils.js" is not permitted.

[FordSubmariner]

I changed the line in tactic.conf from:

Alias /context    /opt/tactic/tactic/src/context

to

Alias /context      /home/apache/tactic/src/context

I reloaded and restarted apache and am still seeing the same problem (unable to load several .js files).

I do have Options FollowSymLinks:
# kf commented out on 02020 May 18: <Directory "/opt/tactic/tactic" >
# <Directory "/home/apache/tactic" >
<Directory "/home/apache/tactic_src_4.7.0.b02" >
    Options FollowSymLinks
    AllowOverride None
    # Apache < 2.4
    #Order Allow,Deny
    #Allow from All
    # Apache 2.4
    Require all granted
</Directory>


I even replaced (as you see above) the symlink /home/apache/tactic with its target. This also did not help.

I'll have more time to check apache logs this weekend and will update here with further progress then.

Thanks again for all your help!

[remkonoteboom]

The logs should give some more information. From what I see, that you have looks correct. Note that this is the apache direct that you do want to use: <Directory "/home/apache/tactic" >

It may also be file permissions. The user www-data is not usually a user you can log in to, but you can temporarily change that in the /etc/passwd file:

For example (for an apache user):

apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin

you can switch it to:

apache:x:48:48:Apache:/usr/share/httpd:/bin/bash

This will allow you to su from root user:

# su apache
(in your case, it would be www-data).

Then try to navigate to the context folder and see if the user is permitted.