Python module load order in 2024

ikolev

Hi,

We're in the process of adapting the Ornatrix plugin to the C4D 2024 API.
After getting the code to compile we ran into an unexpected problem.

The Ornatrix DLL depends on Python and to avoid installing and loading another copy of Python it relies on C4D loading its own Python module before loading Ornatrix.

This so far worked properly (until 2023). In projectdefinition.txt we have python.framework listed under APIS, which I suppose tells C4D that this plugin depends on the Python framework and should be loaded after it. And anyway, on the splash screen the phase "Initializing modules" happens before "Initializing plugins", which makes sense.

However, C4D 2024 loads the Python module after having loaded the plugins. Loading the Ornatrix plugin fails, because python311.dll isn't on any DLL search path (naturally), and hasn't been loaded yet.

So far I can't find any reason why does this happen and how to fix it.
Adding MAXON_DEPENDENCY_ON_MODULE( "net.maxon.python.framework" ) doesn't make any difference.
I just found there was a new C4D version released (2024.0.2) and installed it, but it's still the same.

One really weird detail is that python311.dll is still not found if I add its directory to PATH before running C4D from Command Prompt. Could that mean that C4D changes the PATH variable before loading plugins?

Any ideas how to approach this problem?

Thanks,
Ivan

m_adam

Hi @ikolev is there any reason why you prefer to link directly to the python311.dll instead of using the python.framework? Cause this framework serves as an abstraction layer of the python dll/API so it's version independent and also adds Maxon API memory management so you do not have to care anymore about DecRef/IncRef python objects.

With that's said you can call DLLInterface.AddDllPath and then load the dll.

Cheers,
Maxime.

ikolev

Ornatrix has a core that does not depend on the 3D application host, it exposes many features to Python using pybind11, we can't switch that to a Maxon-provided Python wrapper.

Where can this DLLInterface::AddDllPath function be called from? It has to be called before C4D has tried loading the plugin DLL.
I can only think of creating another plugin that gets loaded before Ornatrix, just to make the call, but DLL loading order is OS-specific and cannot be controlled (I think I've seen threads about this here on this forum).

m_adam

Call this function just before Loading the DLL and it should work.

ikolev

Ornatrix.xdl64 doesn't load python311.dll explicitly, it depends implicitly on it.
When C4D tries to load Ornatrix.xdl64, it fails because of the missing dependency. BTW, here's a piece of the log:

...
 motioncam.xdl64 with module net.maxon.c4d.motioncam
 motiontracker.xdl64 with module net.maxon.c4d.motiontracker
WARNING: Error loading file:///C:/Program Files/Maxon Cinema 4D 2024/plugins/Ornatrix/Ornatrix.xdl64: Could not load dll because of
 Could not load python311.dll due to Windows System Error #126: The specified module could not be found. [win_dll.cpp(313)].. (file:///C:/Program Files/Maxon Cinema 4D 2024/plugins/Ornatrix/Ornatrix.xdl64) [win_dll.cpp(333)]
 Cause: Windows System Error #126: The specified module could not be found. [win_dll.cpp(327)] [general.cpp(513)]
 OrnatrixArnoldTranslator.xdl64 with module com.ephere.ornatrixarnoldtranslator
 pbd_simulations.xdl64 with module net.maxon.pbd_simulations
 python.xdl64 with module net.maxon.c4d.python
 redshift.xdl64 with module net.maxon.c4d.redshift
...

Ornatrix.xdl64 doesn't get a chance to execute a single line of code, it doesn't load at all.
It is possible to split the Python-dependent part of the code into a separate DLL that gets loaded implicitly by the main DLL, but this would be a lot of work.

If, however, Maxon declare that the new behavior of 2024 is intentional and is not going to change, i.e. the Python framework will always load after the plugins, then I suppose we'll have no choice but go for this major reorganization, as I currently don't see any other solution.

I just had another idea: copy python311.dll into plugins\Ornatrix\res\libs\win64 (where we also deploy a couple other DLLs the plugin implicitly depends on).
First, this is a bad idea, second, C4D crashed at startup.

m_adam

Ok now I get it I was missing that you do not load the lib yourself. I will have a look at it Monday.

Cheers,
Maxime.

m_adam

Hi @ikolev previously there was a config module that loaded the Python dll, with 2024.0 we cleanup that and therefor do not need it anymore. The module loading order is

All modules needed for the splash screen.
All ".module" modules (Futurama-only) and core module (including ".config.module" modules)
Remaining .config modules
Remaining non-config non-Futurama modules

The python.config.module was loaded as part of 2., and it loaded the DLL and added it to the DLL path. So the 3rd party module (loaded at 4.) could load the DLL. Loading for same level happens in alphabetical order. So it may helps if you rename your module so that it's loaded after python.
But since the internal python module is not loading anymore the DLL explicitly it's not sure that it will work.

If this does not work, then you will need to create a "ornatrix.config.module" which will load the dll. Find bellow the code that was used.
main.h:

#ifndef MAIN_H__
#define MAIN_H__

#include "maxon/url.h"
#include "maxon/basearray.h"

extern "C" MAXON_ATTRIBUTE_DLL_PUBLIC void QueryStartupOrder(maxon::BaseArray<maxon::Url>& dllsToLoad);

#endif // MAIN_H__

main.cpp:

#include "maxon/configurationdatabase.h"
#include "maxon/dll.h"
#include "maxon/application.h"
#include "maxon/string.h"
#include "main.h"

#if defined(MAXON_TARGET_LINUX)
#include <dlfcn.h>
#endif

#include "./../../../frameworks/python.framework/source/maxon/cpython_raw.h"

static void PrivateQueryStartupOrderDll(maxon::BaseArray<maxon::Url>& dllsToLoad, maxon::Int32 pyVersion)
{
	using namespace maxon;

	iferr_scope_handler
	{
		return;
	};

	String pyLibraryName;
	Url name = (Application::GetUrl(APPLICATION_URLTYPE::RESOURCE_DIR) + "modules"_s + "python"_s + "libs"_s) iferr_return;

#ifdef MAXON_TARGET_WINDOWS
	pyLibraryName = FormatString("[email protected]", pyVersion);
	String pyDirName = FormatString("[email protected]", pyVersion);
	name.Append(pyDirName) iferr_return;
	name.Append(pyLibraryName) iferr_return;
#elif defined(MAXON_TARGET_MACOS)
	name.Append(FormatString("[email protected]", pyVersion)) iferr_return;
#elif defined(MAXON_TARGET_LINUX)
	name.Append(FormatString("[email protected]"_s, pyVersion)) iferr_return;
	name.Append("lib"_s) iferr_return;
	name.Append(FormatString("libpython@[email protected]", pyVersion / 10, pyVersion % 10)) iferr_return;
#endif

	// nothing to do if Python is part of the OS
	if (pyLibraryName.IsEmpty() == true)
		return;

	MAXON_SCOPE
	{
		if (name.IoDetect() == IODETECT::FILE)
		{
#if defined(MAXON_TARGET_DEBUG) && defined(MAXON_TARGET_WINDOWS)
			MAXON_WARN_MUTE_UNUSED DllInterface::AddDllPath(name.GetDirectory());

			finally
			{
				MAXON_WARN_MUTE_UNUSED maxon::DllInterface::RemoveDllPath(name.GetDirectory());
			};
#endif

#if defined(MAXON_TARGET_LINUX)
			int mode = RTLD_GLOBAL | RTLD_NOW | RTLD_DEEPBIND;
			iferr (DllInterface::LoadDll(name, true, false, mode))
#else
			iferr (DllInterface::LoadDll(name, true, false))
#endif
			{
				CriticalOutput("Loading of Python DLL failed: @", err);
			}
			else
			{
				// all done - DLL was pre-loaded
				return;
			}
		}
	}

	// If the python dll could not be loaded remove python module(s)
	for (Int index = 0; index < dllsToLoad.GetCount(); index++)
	{
		String str = dllsToLoad[index].GetName().ToLower();
		if (str.StartsWith("python"_s))
		{
			MAXON_WARN_MUTE_UNUSED dllsToLoad.Erase(index);
			break;
		}
	}
}

extern "C" MAXON_ATTRIBUTE_DLL_PUBLIC void QueryStartupOrder(maxon::BaseArray<maxon::Url>& dllsToLoad)
{
	using namespace maxon;

	// we need to preload python because python.xdl64 statically links the library and it must be 
	// available before c4d can load it

	PrivateQueryStartupOrderDll(dllsToLoad, 310);
}

Cheers,
Maxime.

ikolev

Thank you very much, we'll try this.

ikolev

Thank you again, it worked. I just had to take into account that the python310.win64.framework directory is now named just win64 in Cinema 4D 2024.

And I had to add the path to ai.dll too, because our Arnold translator had the same problem, the C4DtoA module no longer loads ai.dll on startup.