Reimplement core functionality of DIRECT in C++ codebase

[rdb]

There is a consensus that the direct package promotes undesirable programming paradigms, such as the usage of built-ins, god objects, lack of separation of concerns, etc. Furthermore, it creates a dichotomy between the C++ and Python interfaces of Panda3D, meaning C++ users do not get the same quality experience as Python users (see #1518).

There is no intention to deprecate direct, or entirely replace all its features, but the end goal is to transform it so that it isn't a necessary component of Panda3D, but rather pulled in by existing codebases, applications that need its features, or by people who prefers its style. This requires that the most-used features from it are reimplemented in the C++ codebase of Panda3D. We can use this opportunity to address design issues in these interfaces that cannot be fixed without breaking compatibility.

I would generally be in favour of increasing the modularity of Panda3D, and shy away less from moving parts to different components. This generally decreases the surface area of maintenance and different parts can have different release cycles/maintainers, without necessarily making them less "official". It is easy to pip install a sub-component, but hard to exclude something unneeded from the installation. We can still offer those components as part of the installer.

This issue is not to start a discussion about the merits or future status of direct (there can be another issue about that if needed), or an issue about improving direct, but rather a meta-issue to discuss possible alternatives for its core functionality. I would encourage people to branch out with new issues to discuss the individual action points.

If I've missed something important, let me know.

ShowBase

The primary function of ShowBase is to provide an easy interface to:

1. Create the default graphics pipe
2. Create a graphics window
3. Create several display regions, cameras, scene graphs corresponding to each other in the window
4. Set up the data graph for firing off events in response to window interaction
5. Provide access to many singletons and managers
6. Provide assorted utility functions

This is all stuffed into a single god class without proper separation of concerns and the interface becomes unwieldy quickly when you want to deal with eg. multiple windows. In my view, this functionality should be provided by new, proper, hierarchical abstractions created in the C++ codebase (I'm imagining an App/Window/Layer hierarchy), each responsible for its own task. Care should be taken that the interface is still very easy to use, or that we provide still options for rapid prototyping, without compromising on sound architectural design.

We should think about how important the data graph is to Panda and bring it in under the same interface or whether we want to replace it with a simpler system.

Access to assorted singletons is out of scope, and it only creates the problem that ShowBase ends up depending on and pulling in many unneeded components. Those should instead be imported from their appropriate locations.

• Create an issue considering the merits of the DataNode
• Write up the requirements for an application framework
• Propose a design for a new application framework

Loader

We have a Python loader class, which is a thin wrapper around a C++ loader class that is almost as easy to use. We can improve the interface of the C++ loader class where necessary. We could possibly consider an "async-first" design.

• Evaluate the C++ loader interface and what it is missing to be as dev-friendly as the Python Loader

Task

The task system is nowadays a thin wrapper around the C++ AsyncTask system.

• Make minor tweaks to C++ task interface so that its interface is similar to the Python task interface.

Audio3DManager

This is a thin utility class for automatically managing an association between NodePaths and sounds. Let us consider reimplementing such a feature in C++. An idea is to create an AudioNode class that sits in the scene graph, or an AudioEffect that is applied to a node.

• Design a new C++-based class for managing associations between scene graph and sounds

Actor

The Actor interface is a fairly thin wrapper around Character, mostly responsible for managing dictionaries of AnimControl objects to handle multi-part and multi-LOD actors. Consider whether multi-part actors really need to be managed under a single interface, and how we can improve the Character interface to provide the same ease-of-use as Actor.

• Evaluate what the Character class needs to be dev-friendly

Event system

The Python event system (Messenger, DirectObject, EventManager) can be made redundant with minor improvements to the C++ EventHandler and throw_event interfaces.

• Improve the C++ event classes

Python utility classes

Classes like BulletinBoard, things in PythonUtil, etc. are poorly documented and provide very little utility to most Panda3D users, they can stay in direct for users who have interest in it.

Inspection tools

The DIRECT tools, BufferViewer, tkpanels, inspect(), etc. are useful tools, but many of them have fallen into disrepair. I would propose creating a new suite of debugging tools, either implemented as a Python library but preferably on the C++ side, with a "debug overlay" UI that provides various tools to pause the task manager, inspect objects, see buffers, etc. This could also be done as a separate (community) project.

I don't see this as a blocker, though, and it's reasonable to pull in direct as a dev-time dependency to get access to these tools if desired.

Python VFS integration

stdpy and VFSImporter deal with Python integration of important C++ features and should be kept as part of the core Panda3D, and moved to Python submodules of the panda3d package.

• Move stdpy interfaces to panda3d tree
• Reimplement VFSImporter based on current Python importer interface

Distribution tools

These tools should either be moved under panda3d.dist, or separated out into a separate tool that sits out-of-tree. While we do this, we should also evaluate dropping the dependency on setuptools, which is no longer the favoured build system for Python applications, and instead giving it its own entry point and making it compatible with PEP 621 (pyproject.toml).

Filter

Filter provides functionality that a self-respecting 3D engine cannot go without, however it's Python-only, generates (deprecated) Cg code and the customizability leaves something to be desired. Ideally there should be a new, powerful, chainable render pipeline framework implemented in C++, but this is probably a tall order in the short term. Moving it to the panda3d tree would (in my eyes) imply a long-term commitment to the existing interface. Is it okay to keep it in direct in the short term? Or as an entirely separate library? Discussion welcome.

DirectGUI

We can't go without a GUI system in the core - however, DirectGUI interface is notoriously awkward, and implementing a new system from scratch is a large amount of work. I think it is best to improve the C++ PGui interface, which is mature implements virtually all of the functionality of DirectGUI.

• Make PGui nice #1603

Interval

The interval system is one of the most useful components of DIRECT. It is partially implemented in C++ and partially in Python. We could consider moving it to the C++ codebase.

However, I personally find it awkward that it is entirely separate from the animation system, with even the control methods (start/stop) being different from AnimInterface. If we add object-level animations (#457), the overlap of functionality becomes even more awkward.

My vote would be for keeping it in direct in the short term, and in the medium term consider making the animation system in Panda a bit more generic so that intervals can be reimplemented on top of it without too much effort.

FSM

FSM is a very useful utility. It would be nice to have this in C++, but I would say it's at most a medium priority to create a C++ replacement, as it is very easy to implement an FSM yourself.

• Design a C++ based FSM class.

Particles

This is a wrapper around a C++-based particle system. There are some design issues such as relying on evalling files read from disk. It is also not very modern, a GPU-based particle system should be preferred. Let's keep this in direct and consider improving or replacing the C++ particle system as a separate issue.

Distributed / DC parser

We can keep as part of direct for now, but it might be interesting to consider separating it into a third library so that it can be used by direct-less applications; especially as it does not really have a large dependency on the rest of direct other than the event system.

SmoothMover

It's used by distributed and could be kept along with it, but it may be generally useful, in which case it should be moved to grutil.

• Figure out how to use SmoothMover and whether it has other use cases.

Motion Trails

This is generally useful and in relatively good condition, after the recent improvements.

• Patch up C++ motion trail interface to fill any gaps that are provided by the Python interface.
• Move CMotionTrail to grutil.

Level Editor

In poor state, keep in direct. If someone wants to improve it, it is suggested that it be branched off into a separate project.

Cluster client / server

Not generally useful, keep as part of direct or remove.

DirectD

I have no idea what this even is - keep as part of direct or deprecate/remove.

directdevices

Status unknown, possibly obviated by the new input system in 1.10.0 - keep as part of DIRECT.

DirectNotify

The Python-based DirectNotify interface is largely redundant, we already have a C++ Notify system.

• Improve C++ notify interface to be usable from Python.

extensions_native

This contains many deprecated methods, but some useful methods that could be reimplemented in C++, possibly as extension methods if needed

• Reimplement NodePath::analyze() in C++, which requires fixing the SceneGraphAnalyzer dependency issue.
• Create C++ based task or thread for managing an HTTPChannel request.
• Implement a C++ based solution for collision octreefying.
• Consider keeping or deprecating attachCollisionRay and friends.
• Consider reimplementing or deprecating posInterval, etc, which are easily replaced with direct use of LerpPosInterval - also see part about intervals, above.

Tk/wx integration

This code is fragile and not useful to a majority of Panda users - let's keep it in direct, but we can reconsider at a later point.

[rayanalysis]

Perhaps assigning a rough time estimate to each of these development-items might be useful, to get a feel for overall scope. Something like "Interval -- 8 full time weeks, 17,000 lines of code refactor" or something like that. I understand time estimates like that can be pretty rough hewn and depend on developer familiarity with the surrounding source.