Axceleon
Axceleon Home Corporate Info Products Solutions Axceleon Partners Downloads Support from Axceleon News Updates Contact Axceleon
   
[BACK]
Troubleshooting Image Rendering

Troubleshooting EnFuzion Control Root

Troubleshooting EnFuzion Compute Node

Troubleshooting Image Rendering

EnFuzion for Render Farms fails to connect to the Control Root
Select Tools:Test Control Root Access. This opens a dialog for testing the connection. Press Test Connection button.

If the test is successful, then EnFuzion for Render Farms is connected.

If the test fails, then verify the address of the EnFuzion Control Root in the file <EnFuzion_installation_directory>/config/submit.config.txt.

If the address is correct, then verify that the EnFuzion Control Root is operational and accessible over the network.

My jobs complete successfully, but 'Show Image' does not show any images
The EnFuzion Image Browser uses the Maya fcheck or the XSI flip utility to display images. You must have Maya or XSI installed for this feature to work properly. Fcheck is tried first, flip is tried second. If you wish to use a different viewer, please contact Axceleon at support@axceleon.com

My rendering fails to complete successfully
Select the run with failed jobs and double click on the run to open its Job List window. Check the Show All checkbox in the upper right corner. This will show all job retries, including rescheduled jobs. Select a job which causes problems and right click or Ctrl click on Mac OS X to display the job menu.

Select Show Render Log to view the render log produced by the renderer. Check out the log for any error messages or notices from the renderer that might help to diagnose the problem.

If the render log does not exist or if the log does not contain any indication of the problem, select Show Render Standard Output to view the standard output and Show Render Standard Error to view the standard error. These entries show any messages that are recorded to the standard output and the standard error files by the renderer and other programs.

If Show Render Standard Output and Show Render Standard Error do not help, select Show Render Command. This entry displays the command that EnFuzion used on a Compute Node to render the image. Verify that the command is correct by logging into the Compute Node and typing the command in the Command Prompt. If the command fails, then there is a problem with the rendering application, access to the project files or environment variables on the node. Additional entries in the menu help with the problem identification.

Select Show EnFuzion Job Log to view any additional messages from EnFuzion. This log contains all major events during the job execution as well as all EnFuzion errors for the job. Most common problems that are identified in the log are:

  • the file or a directory is not found:
    The node is unable to access project files. Verify that the project files are shared over the network and accessible to compute nodes.

  • the following user command exited with status code:
    The rendering application failed to complete successfully. Verify that the rendering application is installed on the node, verify the environment variables required by the application and check out the render log, the standard output and the standard error for any error messages.

Select Show Render Environment to view and verify the environment variables that were defined for the renderer. The most common problem with environment variables is that their values are incorrect. If values are incorrect, modify values in the EnFuzion node environment file in <EnFuzion_installation_directory>/environment.txt.

If the problem persists, select Zip Diagnostic Files. This produces the zip file run-<runid>.zip with diagnostic files. The file is created in the <EnFuzion_installation_directory>/submissions/work directory. Send this zip file to support@axceleon.com for assistance.

   

My output images are not correct
Make sure that your scene and your render global parameters are correct by rendering several images locally on your workstation before submitting the scene to your render farm.

If images are correctly rendered locally, but fail to render on the render farm, check out the section on failed rendering.

3ds Max: Using 3ds Max 8 on Compute Nodes
If you use 3ds Max 8 on a Compute Node, make sure that previous versions of 3ds Max on the same machine are uninstalled and their directories deleted.

After Effects: After Effects fails to render on Mac OS X platform
If no user is logged in on the Mac OS X platform, then rendering might fail with the following message:

INIT_Processeses(), could not establish the default connection to the WindowServer

There are two workarounds to avoid this situation:

  • Login the enfuzion user permanently;

  • Set the ownership of the After Effects aerender command to root with the following commands:
    cd /Applications/Adobe After Effects 6.5
    sudo chmod +s aerender
    sudo chown root aerender

CINEMA 4D: CINEMA 4D fails to render
Verify that your scene is prepared for rendering on an EnFuzion render farm. The logging in the scene must be turned on. To turn on the logging, open the scene in CINEMA 4D and select Render Settings:Options, turn on Log File.

If the logging is turned on, but the scene is still not rendering, then verify that your Compute Node is configured for rendering with CINEMA 4D. The enfuzion user on the node must have write, read and modify permissions to the CINEMA 4D directory, so that it can manage the render log. These permissions are turned on as follows:

  • right click on C:\Program Files\MAXON\CINEMA 4D R9 in Explorer;
  • select Properties:Security;
  • under Group or user names:, select Users;
  • under Permissions for Users:, turn on Full Control;
  • click OK.
   

Maya: Maya fails to render on Mac OS X platform
If no user is logged in on the Mac OS X platform, then rendering might fail with the following message:

INIT_Processeses(), could not establish the default connection to the WindowServer

There are two workarounds to avoid this situation:

  • Login the enfuzion user permanently;

  • Set the ownership of the Maya Render command to root with the following commands:
    cd /Applications/Alias/maya7.0/Maya.app/Contents/bin
    sudo chmod +s Render
    sudo chown root Render

Maya: Some Maya frames fail to render on Mac OS X platform
The rendering sometimes fails with a message similar to the one below:
2005-08-14 21:33:45.693 MayaBatch[2781] CFLog (0): CFMessagePort: bootstrap_register(): failed 1100 (0x44c), port = 0x2e03, name = 'Processes-0.7471105' See /usr/include/servers/bootstrap_defs.h for the error codes.
2005-08-14 21:33:45.694 MayaBatch[2781] CFLog (99): CFMessagePortCreateLocal(): failed to name Mach port (Processes-0.7471105) CFMessagePortCreateLocal failed (name = Processes-0.7471105 error = 0)

The solution is to reboot the machine. There is no need to resubmit your Scene as EnFuzion automatically reschedules the failed jobs to other available machines.

Maya: The Thumbnail Browser does not show tiles
This feature is not yet supported. The View Thumbnail feature does not work for frames rendered in tiles.

Maya: Maya frames fail to render on heterogeneous render farms
Your render farm is heterogeneous if one or more Compute Nodes run a different operating system than your Submit Computer.

All external texture and other file references in the scene must be relative to the Maya project. EnFuzion automatically adjusts path values between different platforms when it calls the renderer from a command line. EnFuzion will not change any paths that are specified in the scene itself. If you want to use file paths that are not relative to the project and have a heterogeneous render farm, please contact Axceleon at info@axceleon.com.

Alias mental ray: How do I generate .mi files from a command line or with a script?
The following command takes a Maya scene and generates .mi files:

maya -batch -command "Mayatomr -mi -pf 2 -file <output>.mi -xp rrarrarrrr" -file <scene.ma>

Replace <output> with the name of the output .mi files and <scene.ma> with the Maya scene file.

The command can be included in a script or executed from a command line.

Alias mentay ray: How do I configure compute nodes for Alias mental ray Standalone?
EnFuzion provides a separate script to configure mental ray Standalone. This script is <enfuzion_dir>/enfenv-maya-mi.sh on Linux and Mac OS X and <enfuzion_dir>/enfenv-maya-mi.bat on Windows. This is a shell/batch script that initializes the environment for the Alias mental ray Standalone rendering package. The script includes default installation values for mental ray. If required, modify the values for your configuration.
   

XSI: Image viewer Flip.exe fails on Windows
When you try to use Softimage flip.exe to view images, the program might fail with the following message:

flip.exe - Unable to Locate DLL
The dynamic link library ilcor10.dll cannot be found ...

In this case, find ilcor10.dll on your system and add the directory it is located in to the PATH environment variable on your system or copy ilcor10.dll to the directory with flip.exe.

XSI: Image converter imgconv.exe fails on Windows
When EnFuzion tries to use Softimage imgconv.exe to convert images to thumbnail JPEGs, the program might fail to work.

In this case, make sure that the SI_IMAGE_PATH environment variable is set to <XSI_installation_directory>\Application\bin\sil. Default value for XSI 5.0 on Windows is C:\Softimage\XSI_5.0\Application\bin\sil.

Troubleshooting EnFuzion Control Root

I am unable to connect to the EnFuzion Web
Verify that the URL address in your Web browser is correct. The EnFuzion Web address is:

http://<EnFuzion_root_host>:10101

You should be greeted by the EnFuzion Welcome page.

If the EnFuzion Welcome page is not displayed, verify that EnFuzion Root is running and that the network to the EnFuzion Root host is working.

If the EnFuzion Welcome page is displayed, click on the Cluster link in the header to verify EnFuzion operation.

If the Cluster link returns a General Error page, then verify that the EnFuzion Root is running.

How do I verify that the EnFuzion Root is running?
On Windows, open Task Manager and confirm that the following processes are running: enfDispatcher.exe and enfeye.exe.

On Mac OS X and Linux, execute the following command line:
ps ax | grep enf
Confirm that the following processes are running: enfdispatcher and enfeye.bin.

If the enfeye process is not running, reboot the machine.

If the enfdispatcher process is not running, check out the enfdispatcher section.

The enfdispatcher process is not running on the EnFuzion Root computer
Verify in the Task Manager on Windows or with the ps ax command on Mac OS X and Linux that there is no enfdispatcher process.

If there is no enfdispatcher process, check out the dispatcher log for any error messages. The dispatcher log is located in the directory <enfuzion_installation_directory>/work/enfuzion.log. The most common problems are a missing or invalid EnFuzion license key or no space on the working disk.

If the EnFuzion license key is missing or invalid, install the key.

If your working disk is running out of space with less than a few Mb available, EnFuzion Root software records this in the log and preventively terminates. Delete redundant files and reboot the machine or restart EnFuzion.

If the log file <enfuzion_installation_directory>/work/enfuzion.log does not exist, delete all the files in the <enfuzion_installation_directory>/work directory, verify that the enfuzion user can write to the directory and reboot the machine.

If there is still no enfdispatcher process and no enfuzion.log file after the machine is rebooted, verify that EnFuzion is configured to start at the boot time.

   
How do I configure EnFuzion Root Computer to start EnFuzion at the boot time?
Check out sections for specific computing platforms: Windows, Mac OS X and Linux.

How do I configure EnFuzion Root Computer to start EnFuzion at the boot time on Windows?
On Windows, the default option of the EnFuzion graphical installer is to automatically configure the computer to start EnFuzion at the boot time. To configure the computer manually, select the menu item Start:All Programs:EnFuzion:Control Root Utilities:Register EnFuzion Root service. To start EnFuzion, select the menu item: Start:All Programs:EnFuzion:Control Root Utilities:Start EnFuzion Root service.

If the EnFuzion does not start after the reboot, verify all the steps in the section about the enfdispatcher process not running.

If the EnFuzion still does not start and there is no log in <enfuzion_installation_directory>/work/enfuzion.log, verify that Group Policy is installed on your Windows machine. EnFuzion uses a startup script on the EnFuzion root host to start after a reboot and Group Policy is required for this functionality to work. Execute the following steps to install Group Policy:

  • start the Microsoft Management Console
    • in Start menu, select Run...
    • type mmc
  • add Group Policy Snap-In in the Management Console
    • in Console/File menu, select Add/Remove Snap-In...
    • click Add...
    • select Group Policy
    • click Add
    • click Finish
    • click Close
    • click OK
  • verify the Dispatcher startup script
    • double click Local Computer Policy
    • double click Computer Configuration
    • double click Windows Settings
    • click Scripts (Startup/Shutdown)
    • in the panel on the right, double click Startup
    • verify that the C:\EnFuzion\bin\enfstartup.exe is on the list
    • click OK
  • (optional) specify 10s for time that Windows will wait for a startup script
    • double click Administrative Templates
    • double click System
    • in the panel on the right, double click Maximum wait time for Group Policy scripts
    • select Enabled
    • type 10 in the Seconds field
    • press the key
  • (optional) specify that startup scripts should execute asynchronously
    • in the panel on the right, double click Run startup scripts asynchronously
    • select Enabled
    • click OK
    • exit the Microsoft Management Console
    • in Console/File menu, select Exit
    • click No

How do I configure EnFuzion Root Computer to start EnFuzion at the boot time on Mac OS X?
Verify that the EnFuzion startup files are installed. The files are in the directory /Library/StartupItems/EnFuzion. The directory should include two files, EnFuzion and StartupParameters.plist.

If the directory or the files do not exist, go to the directory with an unpacked EnFuzion distribution and execute:
sudo ./install-service

If the directory and the files exist, verify that EnFuzion is listed in the system startup file. The system startup file is /etc/hostconfig. The file should contain a line:
ENFUZION=-YES-

To start EnFuzion, execute the following command line:
sudo SystemStarter start "EnFuzion Control Root Service"

How do I configure EnFuzion Root Computer to start EnFuzion at the boot time on Linux?
Verify that the EnFuzion startup script is installed. The script is in the file /etc/init.d/enfuzion.

If the file /etc/init.d/enfuzion does not exist, go to the directory with an unpacked EnFuzion distribution, log in as a root user and execute:
./install-service

If the file /etc/init.d/enfuzion exists, verify that EnFuzion is on the list of services that are started at the boot time. Execute the command:
chkconfig --list | grep enfuzion
Output should include several on levels:
enfuzion 0:off 1:off 2:on 3:on 4:on 5:on 6:off

To start EnFuzion, execute the following command line:
/etc/init.d/enfuzion start

How do I install an EnFuzion license key?
EnFuzion license keys are stored in file <EnFuzion_installation_directory>/config/enflicense.txt. Verify that the file exists and that it contains your license key.

If the license key file does not exist, copy your EnFuzion license key to <EnFuzion_installation_directory>/config/enflicense.txt.

If the license key file exists, but the enfdispatcher process is not running, verify that there are no messages about an invalid license in the EnFuzion log file in <EnFuzion_installation_directory>/work/enfuzion.log.

If your license key is invalid then the log file contains messages similar to the following:
Mon Jan 16 18:52:51 2006: CL10102 cluster host3:10102 create port 10102
Mon Jan 16 18:52:51 2006: CL10102 cluster host3:10102 message build 9.0.006 for Linux 2.4.5-3smp i686
Mon Jan 16 18:52:51 2006: CL10102 cluster host3:10102 message reading root options from file "/home/john/enfuzion/config/root.options"
Mon Jan 16 18:52:51 2006: CL10102 cluster host3:10102 message the license file "/home/john/enfuzion/config/enflicense.txt" contains an invalid EnFuzion Root license.
Mon Jan 16 18:52:51 2006: CL10102 cluster host3:10102 message The license is invalid for the following reasons:
Mon Jan 16 18:52:51 2006: CL10102 cluster host3:10102 line 2: day date 16 1 2006, expired 15 12 2005

Verify that the content of <EnFuzion_installation_directory>/config/enflicense.txt matches the content of the EnFuzion license key that you received in your e-mail.

If your EnFuzion license key expired, send a request for a new license key to support@axceleon.com.

For an additional assistance, report the problem to support@axceleon.com and include <EnFuzion_installation_directory>/work/enfuzion.log and <EnFuzion_installation_directory>/config/enflicense.txt in your message.

Troubleshooting EnFuzion Compute Node

How do I verify that the EnFuzion Node in my cluster is working?
Open the EnFuzion Web page in your Internet browser:
http://:10101

Click on the Nodes link in the header to obtain the Nodes table.

Under the Host column, find the host name for the Node and check its Status.

If the node status is Idle, Executing or Busy, then the Node is working.

If the host name is not in the table or if its status is Down, then the Node is not working. In that case, verify that the EnFuzion Node is running on the host.
   

How do I verify that the EnFuzion Node is running on the host?
On Windows, open Task Manager and confirm that the following processes are running: enfStarterSvc.exe and enfNodeServer.exe.

On Mac OS X and Linux, execute the following command line:
ps ax | grep enf
Confirm that the following process is running: enfnodeserver.

If the enfnodeserver process is not running, check out the enfnodeserver process section.

If the enfnodeserver process is running, confirm that the node is not working, using the EnFuzion Web.

If the enfnodeserver process is running, but the node is not working, then the Node is not connecting to the EnFuzion Root. Check out the section on Node is not connecting.

The enfnodeserver process is not running on the EnFuzion Node computer
Check out sections for specific computing platforms: Windows, Mac OS X and Linux.

The enfnodeserver process is not running on the EnFuzion Node computer on Windows
Verify in the Task Manager that there is no enfnodeserver process.

Check out the Starter Service log for any messages. The log is located in the file <EnFuzion_installation_directory>\temp\enfstarter.log. The EnFuzion Starter Service starts the enfnodeserver process on the local host. The Starter Service log reports any errors that happened during that process.

A common problem is an invalid user name or password. The Starter Service log contains the following line:
Thu Jan 12 20:00:41 2006: Logon failed for user 'enfuzion' - Logon failure: unknown user name or bad password.

In this case, create the user on the local machine or provide a correct password. If the Node is defined locally, then the password is stored in the file <EnFuzion_installation_directory>\service.config.txt. If the Node is defined on the EnFuzion Root host, then the password is stored in the file <EnFuzion_installation_directory>\config\enfuzion.nodes.txt.

If you change the service.config.txt file, then you must restart the Node. Stop the Node by selecting the menu item Start:All Programs:EnFuzion:Compute Node Utilities:Stop EnFuzion Node service. Start the Node by selecting the menu item Start:All Programs:EnFuzion:Compute Node Utilities:Start EnFuzion Node service.

If you change the enfuzion.nodes.txt file, then you must restart the EnFuzion Root. Stop the Root by selecting the menu item Start:All Programs:EnFuzion:Control Root Utilities:Stop EnFuzion Root service. Stop the Node by selecting the menu item Start:All Programs:EnFuzion:Control Root Utilities:Start EnFuzion Root service.

If the enfnodeserver process is still not running and there are no error messages in the Starter Service log, then check out the Node Server log. The log consists of two files in the directory <EnFuzion_installation_directory>\temp. Files are called enfnodea.log and enfnodeb.log. EnFuzion uses a circular double buffering to keep the file sizes limited. The file with the latest entries contains the latest reports. The files contain any critical error messages by the Node.

If the problem is not evident from the Node Server log, then send a report to support@axceleon.com and include files enfstarter.log, enfnodea.log and enfnodeb.log.

The enfnodeserver process is not running on the EnFuzion Node computer on Mac OS X
Verify with the ps ax command that there is no enfnodeserver process.

Verify that the EnFuzion Node startup files are installed. The files are in the directory /Library/StartupItems/EnFuzionNode. The directory should include two files, EnFuzionNode and StartupParameters.plist.

If the directory or the files do not exist, go to the directory with an unpacked EnFuzion distribution and execute:
sudo ./install-svcnode <root_host>

If the directory and the files exist, verify that EnFuzion is listed in the system startup file. The system startup file is /etc/hostconfig. The file should contain a line:
ENFNODE=-YES-

To start EnFuzion, execute the following command line:
sudo SystemStarter start "EnFuzion Compute Node Agent"

If the enfnodeserver process is still not running, then check out the Node Server log. The log consists of two files in the /tmp directory. Files are called .enfnodea.log and .enfnodeb.log. EnFuzion uses a circular double buffering to keep the file sizes limited. The file with the latest entries contains the latest reports. The files contain any critical error messages by the Node.

If the problem is not evident from the Node Server log, then send a report to support@axceleon.com and include files .enfnodea.log and .enfnodeb.log.

 

The enfnodeserver process is not running on the EnFuzion Node computer on Linux
Verify with the ps ax command that there is no enfnodeserver process.

Verify that the EnFuzion startup script is installed. The script is in the file /etc/init.d/enfnode.

If the file /etc/init.d/enfnode does not exist, go to the directory with an unpacked EnFuzion distribution, log in as a root user and execute:
./install-svcnode <root_host>

If the file /etc/init.d/enfnode exists, verify that EnFuzion is on the list of services that are started at the boot time. Execute the command:
chkconfig --list | grep enfnode
Output should include several on levels:
enfnode 0:off 1:off 2:on 3:on 4:on 5:on 6:off

To start EnFuzion, execute the following command line:
/etc/init.d/enfnode start

If the enfnodeserver process is still not running, then check out the Node Server log. The log consists of two files in the /tmp directory. Files are called .enfnodea.log and .enfnodeb.log. EnFuzion uses a circular double buffering to keep the file sizes limited. The file with the latest entries contains the latest reports. The files contain any critical error messages by the Node.

If the problem is not evident from the Node Server log, then send a report to support@axceleon.com and include files .enfnodea.log and .enfnodeb.log.

An EnFuzion Node is not connecting to the EnFuzion Root
Verify that there is no active firewall on the EnFuzion Root Computer. If a firewall is required on the EnFuzion Root Computer, then configure EnFuzion to use static port numbers. Perform the following steps on the EnFuzion Root Computer:

add the following line to <EnFuzion_installation_directory>/config/root.options:
jobport 10104

in the firewall configuration, open the following ports:
10101, 10102, 10103, 10104;
restart EnFuzion on the Root Computer.

Verify in the Task Manager on Windows or with the ps ax command on Mac OS X and Linux that the enfnodeserver process is running.

If the process is not running, check out the section on Node is not running.

If the enfnodeserver process is still not running, then check out the Node Server log. The log consists of two files. On Windows, files are called enfnodea.log and enfnodeb.log and are in the <EnFuzion_installation_directory>\temp directory. On Mac OS X and Linux, files are called .enfnodea.log and .enfnodeb.log and are in the /tmp directory. EnFuzion uses a circular double buffering to keep the file sizes limited. The file with the latest entries contains the latest reports. The files contain any critical error messages by the Node.

If the problem is not evident from the Node Server log, then send a report to support@axceleon.com and include the files enfnodea.log and enfnodeb.log.