Is there a way to speed up tactics?

[kanta]

I am sorry, kanta.  I mean to reply to your post, but accidentally  overwrite it.  There is no way to undo.  At any rate, my response is repeated here:

I also tried'load balancing'.

What was the result of this?  It should look something like the attached.

Nothing happens when I hit the'load balancing' button.

There is no text like the image you attached.






[root@q361-0706 ~]# top
top - 14:13:53 up 80 days, 21:58,  1 user,  load average: 0.62, 0.36, 0.24
Tasks: 159 total,  2 running, 157 sleeping,  0 stopped,  0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  : 88.8 us, 11.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16086852 total,  3333240 free,  2600172 used, 10153440 buff/cache
KiB Swap:  4194300 total,  4178940 free,    15360 used. 12207224 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+ COMMAND
7638 tactic    20  0 2928376 705844  9624 S 100.0  4.4  1234:05 python3
7456 tactic    20  0 2998552 839300  9596 S  1.3  5.2  1240:44 python3


I think the reason of slow Tactic is Tactic process which executed by python3 /opt/tactic/tactic/src/bin/startup.py 8082 reached to CPU 100%. Is there possible to be slow due to the carrying out Tactic inner code?

[remkonoteboom]

I am not sure what that python3 process is doing.  But, it is a clue.

1) Does it stay at 100% always or just when you pressed load balancing?
2) Are others using the system while your are looking at this?

If 100% on that single CPU is continuous, that means everyone, including your load balancing test, is bottlenecked through a single CPU.  My guess, at the moment, is that load balancing is not on.  If this is the case, then, you are not utilizing all of your CPUs and would explain the slow behavior.

Can you do:

# ps -ef | grep python

Which produces something like:

tactic    335 24850  0 15:08 ?        00:00:01 python3 /opt/tactic/tactic/src/bin/startup.py 8081
tactic    765 24850  0 15:10 ?        00:00:01 python3 /opt/tactic/tactic/src/bin/startup.py 8082
tactic    1078 24850  0 15:11 ?        00:00:01 python3 /opt/tactic/tactic/src/bin/startup.py 8083
root    24843    1  0 Apr21 ?        00:00:00 su - tactic -m -c /usr/bin/python /opt/tactic/tactic/src/bin/monitor.py

Also, can you post the load balancing part in the TACTIC apache conf file, which is in: /etc/httpd/conf.d/tactic.conf

It should be something like:

# Using the ProxyPass directives
<Proxy balancer://tactic>
BalancerMember http://localhost:8081/tactic
BalancerMember http://localhost:8082/tactic
BalancerMember http://localhost:8083/tactic
</Proxy>
ProxyPass /assets !
ProxyPass /context !
ProxyPass /doc !

ProxyPass /tactic balancer://tactic/
ProxyPass /projects balancer://tactic/

ProxyPass / balancer://tactic/

[EricTsuei]

I am running at a 104 core 64GB server, using windows.
the speed seems same as your speed. it's really slow.
I also increase the balancer member to 40, it's better while you have multiple users. but it used so much memory.
but each user load pages is still slow. but it's more stable with more balancer member.

[remkonoteboom]

Is that correct? 104 Cores? And with 40 load balanced processes? That is a huge amount. I know you have a lot of projects going, but this is still quite a lot of processes.

Would you be able to post a view of the Windows Task Manager on the server? I would really like to know what the bottleneck may be.

[EricTsuei]

Is that correct?  104 Cores?  And with 40 load balanced processes?  That is a huge amount.  I know you have a lot of projects going, but this is still quite a lot of processes.

Would you be able to post a view of the Windows Task Manager on the server?  I would really like to know what the bottleneck may be.

sure,
I am now running in ESXi virtual machine, and assign 96 core to windows server, save 8 core for stable running ESXi.
   
   
   
   

[remkonoteboom]

To be honest, I have never seen an installation that required so many processes. This includes full scale CG feature films and multiple simultaneous full episodic television projects. Of course, you may have a very large number of users that simultaneously hit the system or are running large number of automated processes.

When under load, how many of those have python processes at 100%?

[EricTsuei]

NO, I am just doing this to see if this can help improve user load speed.
but, seems it's more stable, but won't get more speed.
and we are not so large scale studio, but we are producing 2 CG feature films and 4 other TV episodes.

[kanta]

According to your help, the speed of Tactic problem has solved.

The problem was considered due to the speed of Tactic engine, however, it was setting issue. Current loading speed is just 2sec after modifying setting. (previous speed was 15sec)

I really thank you for your help.

I shared this problem and solution if it would be useful to other users who has same issue.


------------------------------
After modification (there was no change in the computer hardware)




-----------------------------------------

Modification


1.
the Python processes are running for a long time , and it's causing sluggishness. We set the process_time_alive setting to 30 in /opt/tactic/tactic_data/config/tactic-conf.xml to prevent this in the future:
<process_time_alive>30</process_time_alive>
It means the processes will be refreshed every 30 minutes automatically. Users will not notice.



2.
We also increased load-balancing to 6 TACTIC processes (previously it was 3) since your machine had enough capabilities.