MAPS

##################################################################################
__ __ _____ _____
| \/ | /\ | __ \ / ____|
| \ / | / \ | |__) | (___
| |\/| | / /\ \ | ___/ \___ \
| | | |/ ____ \| | ____) |
|_| |_/_/ \_\_| |_____/

For
Mouse Audio Process Streaming

Scenario interaction mouse/audio
Hugo Baranger aka HV$VH & Alexandre Sune aka Mr_H4l3x ##################################################################################

#1 The general idea:

On a computer, sound is generated in association with the mouse 'click' event.
Click is the trace of the human interaction with the machine.
If we can record in a database all the clicks done on a specific computer we can produce
an history of the human interactions.
This database of interactions is used to generate an audio composition.

#2 How to do that:

A/ Capture each "click" events, and record it in a database.

To capture each "click" we can use Xbindkey, XDOTOOL and a bash script associated.
The datas are recorded in a SQL database with the same script.

B/ Generate audio.

The database is dumped in a file,
and each line of the file is interpreted by a PureData Patch to produce audio synthesis.
At the end of the file the database is dumped and the content of the file is replaced
by a new and fresh one.

This will run forever.

C/ Stream the audio composition in realtime.

The computer will run has a webserver and use icecast2 to perform Live Streaming.
The Puredata Patch will use the element "MP3cast~" from the Library "Unauthorized"
to connect audio of the Computer to the Listening Port of Icecast2.

#3 the project is divided in 3 Blocs.

Bloc #1 capture mouse event and record it
Bloc #2 interprete recorded event to produce audio content
Bloc #3 Stream it Live Baby!

-----------------
| BLOC #1 |
-----------------

At the very beginning i was thinking about « XDOTOOL » to manage this part, with a little bash script and « voilà! »… But, after many try I came to the conclusion that this method slow down all the computer’s running process, because the script have to look at « click » events and record these in the same time, hum, not so easy to do… but keyloggers exist so there will be a way to do it, maybe a sneeky one.

The Python way:
Let me introduce you « pynput » a library that allows you to control and monitor input devices.

Here is the script used:

#import the library for listening inputs and date/time
from pynput.mouse import Listener
import datetime

#function used when  a « click » is detected 
#1 check the date/time && get the mouse position (x y)
#make a string with theses informations
#record it in a text file
def is_clicked(x, y, button, pressed):
	if pressed:
		d=datetime.datetime.now()
		dd=d.strftime("%d %m %Y %H %M %S")
		finalString=format(dd)+" "+format(x)+" "+format(y)+"\n"
		theFile=open("/home/pauline/theMouseRecord.txt","a", encoding="utf-8")
		
		theFile.write(finalString)
		theFile.close()
		
		
#a simple listener to trigger the clicked mouse 		
with Listener(on_click=is_clicked) as listener:
	listener.join()

Now we just have to launch the script at startup and it will be good :) (it's done from PureData with the "shell" object, more info bellow)

Ok, stage completed ✅

-----------------
| BLOC #2 |
-----------------

To make some noise, we need to use a versatile programmation language: PudeData.
I will let Hugo explain more about the sound research and work, and for now i just want to explain the very basis of the patch. We use PureData vanilla and some useful library:
"ggee" to get the "shell" object (ability to send bash commands directly from PureData)
"Unauthorized" to get the "MP3cast~" object (ability to stream audio via icecast2 from PureData)
"Zexy" to have access to a nice collections oaf objects (very useful for Hugo)

We need to read a text in PureData line by line and extract informations (date/time/x/y), so I used to launch a simple bash script from PureData using "shell" object.

The bash script:

#!/bin/bash

input="path/to/the/file.txt"

while IFS= read -r line
do
  echo "$line"
  sleep 1s
done < "$input"

Once launched the script will read and output the txt line by line and at the end the "shell" object will notice us with a "bang" like this we can re-launch the bash script when the PureData process is done.

Ok, PureData can read a txt file ✅

PureData can play sounds now ... (insert Hugo part here)

Dear PureData, You can read, you can sing, can you stream?
For audio the streaming process we need the "MP3cast~" object correctly configured (portnumber/password) on localhost. With Icecast2 running...

We need another bloc.

-----------------
| BLOC #3 |
-----------------

Icecast2 is a very nice solution to stream audio on the web, we use it with Hugo for more than a year on another project (a generative web radio "ASH.b-22" based on number radios). Icecast is very stable and reliable, but it's a pain in the ass to configure SSL on it. Because Icecast need a specific file made with "fullchain.pem" and "privkey.pem"

You can do it like this:

sudo cat /etc/letsencrypt/live/YOUR-DOMAIN-NAME/fullchain.pem /etc/letsencrypt/live/YOUR-DOMAIN-NAME/privkey.pem > /etc/icecast2/bundle.pem

Once it's done, you need to add a specific port for SSL in the Icecast.xml:

    <listen-socket>
        <port>8080</port>
    </listen-socket>
   
    
    <listen-socket>
        <port>8443</port>
        <ssl>1</ssl>
    </listen-socket>

And add the path of the SSL bundle:

<ssl-certificate>/etc/icecast2/bundle.pem</ssl-certificate>

We keep the no-SSL one still open, because PureData's "MP3cast~" object can't stream on ssl (the stream from PureData has a localhost target so it's not so problematic 😬 ) maybe it will be useful to make this modification on "MP3cast~" with the sources... ;)

Once the Icecast2 server is running, we need to add an audio element linked to the stream in the no-VNC webpage of SUDA.

And "voilà!" 😅

Mr_H4l3x