Updating the scheduling library

[gparmer]

Summary: sl and the scheduling infrastructure are showing their age and the fact that we've learned a lot since the initial implementation. This issue is meant to be a list of desired updates. @phanikishoreg is doing a lot of work on this, so now is the time to spruce things up.

• Allocation. The current plan for controlling how data-structures (read: threads, and block-points) are allocated is to enable an API that is polymorphic for multiple implementations. We realistically only use a static backend now. Another option is to pass the memory into the library to use for these structures. This would enable the memory properties of the system to be defined by the environment in which sl is embedded. There are possible problems with this:

  1. Deallocation is hard. If we ask to de-initialize a thread, but the library says this operation cannot be done now (i.e. that thread is executing), then when will the surrounding system know when it can be deallocated?
  2. blkpts need to be intergrated into whatever story is required for the memory allocation/free story.

• Timing policy. The timing policy is not at all configurable currently. This needs to be designed into the system. If we do maintain a heap of timeouts, then we need to understand how large it must be, and where to get the memory from.

• Thread ids. The thread ids need to be allocated by the library, and not by the kernel. Then we need to replace the introspection kludge to get the thread ids.

• Hierarchical scheduling. We need to collect the complexity in text in the discussion here. There have been too many hacks in the past around this, and we need to understand why to do a re-design.

• Asynchronous operation optimization. I'm not going to detail this here, but we implemented the relationship between asnd and rcv far too defensive. We can optimize by getting rid of this.

• What else?

@phanikishoreg is the prime on the scheduler, @WenyuanShao likely should start to think about some of this, and @hungry-foolish also interacts with the scheduler.