Demo question

[skelder]

Hello,

I have few questions about demo

1. how can i use my own music for demo?

2. which get_demo.py is correct one? there are multiple of them on the root folder as well as under Demo folder

3. I get similar bug as in Demo bugs #3 . I added audio folder in demo folder, but i get same error for output folder now

Thank you

[xrenaa]

Hi,

Sorry for the late reply.

1. You can use your own music. You just need to read your own music file (Mono is required) and reshape it to (XX,50,1600). Then you can input it into your model. I remember the range of music is not normalized. If you have any difficulties, I may add a Jupiter notebook when I am free.

2. The file in root dir is current. Those in the Demo folder are used for writing my paper.

3. Can you show me your error?

[skelder]

Thank you for clarification!

For some reason the demo works correctly without any bugs now when i try with root folder get.domo.py.
I did do clean reinstall of the environment, maybe that helped.

Now that i got results from the demo i have some more questions:

1. what is the difference between the upper skeleton and lower skeleton on the resulting videos?
   is the upper one generated and lower one baseline?

2. If i have bunch of dance videos that i want to train upon, all i have to do is run OpenPose on them and get the skeleton data for each frame?
   Is there any more post processing involved to convert the OpenPose result to one that can be used by your algorithm?

Thank you!

[xrenaa]

Hi, the upper and the lower is ground truth and generated. And you can refer to ./dataset/dataset_usage.ipynb to see the details of the Json file I provide. And you just need to extract the skeleton and make the music and the skeleton in pair.

[skelder]

Hi again.

I managed to train the algorithm on my own dataset.

Now i have few questions:

1. i noticed that the end result is confined to some video resolution.
   I was originally using 1280x720 videos for training then reduced them to 500x500 to correctly position the target person in the frame.
   What resolution were your training videos?

2. My current result has weird slowed down sound in the videos.
   Should i match the sampling rate of my audio to the audio in sample dataset?
   What was the sampling rate for those audio?

3. when using the openpose did you use COCO model?
   I assumed so because of the keypoint count

4. how can i make this algorithm to generate video given a music that was not included in the dataset?

Thank you!

[NuhaNasser]

Hello @skelder,
I am trying to construct my own dataset, and train the model on my dataset.
May you please share your code for constructing the dataset, and how did you modified the code according to your dataset?
My email is: [email protected]
Any help will be appreciated.
Thank you in advance,
Nuha

[Ha0Tang]

Hi, what does 1600 mean?

[xrenaa]

Hi, what does 1600 mean?

16000 hz per second and we have 1600 hz per 0.1 second.

[aryanna384]

Hi,

Sorry for the late reply.

1. You can use your own music. You just need to read your own music file (Mono is required) and reshape it to (XX,50,1600). Then you can input it into your model. I remember the range of music is not normalized. If you have any difficulties, I may add a Jupiter notebook when I am free.
2. The file in root dir is current. Those in the Demo folder are used for writing my paper.
3. Can you show me your error?

Hello, I got the same question about running my own music, can you explain it in detail please?Thanks!

[xrenaa]

Hi,
Sorry for the late reply.

1. You can use your own music. You just need to read your own music file (Mono is required) and reshape it to (XX,50,1600). Then you can input it into your model. I remember the range of music is not normalized. If you have any difficulties, I may add a Jupiter notebook when I am free.
2. The file in root dir is current. Those in the Demo folder are used for writing my paper.
3. Can you show me your error?

Hello, I got the same question about running my own music, can you explain it in detail please?Thanks!

Hi,
I think you can just read the audio file with frequency 16khz. For example, the 5s clip will have 50 * 1600 hz and then you can just resize it to (xx, 50, 1600). Then it can be the input to the model.

[zhangjiwei-japan]

could you give me some audio samples?thanks

[xrenaa]

could you give me some audio samples?thanks

Hi, I think you can use the samples of the json files as I provided. Or I think you can use your own music loaded with librosa.