Skip to main content

MAPS

##################################################################################
__  __          _____   _____ 
|  \/  |   /\   |  __ \ / ____|
| \  / |  /  \  | |__) | (___
| |\/| | / /\ \ |  ___/ \___ \
| |  | |/ ____ \| |     ____) |
|_|  |_/_/    \_\_|    |_____/
                                
    For 
Mouse Audio Process Streaming

Scenario interaction mouse/audio                      
Hugo Baranger aka HV$VH & Alexandre Sune aka Mr_H4l3x ##################################################################################

#1 The general idea:

On a computer, sound is generated in association with the mouse 'click' event.
Click is the trace of the human interaction with the machine.
If we can record in a database all the clicks done on a specific computer we can produce
an history of the human interactions.
This database of interactions is used to generate an audio composition.

#2 How to do that:

A/ Capture each "click" events, and record it in a database.

The record would be formated like this:
ID | DAY | MONTH | YEAR | Hour | Minute | second | positionX | position Y

To capture each "click" we can use Xbindkey, XDOTOOL and a bash script associated.
The datas are recorded in a SQL database with the same script.

B/ Generate audio.

The database is dumped in a file,
and each line of the file is interpreted by a PureData Patch to produce audio synthesis.
At the end of the file the database is dumped and the content of the file is replaced
by a new and fresh one.

This will run forever.

C/ Stream the audio composition in realtime.

The computer will run has a webserver and use icecast2 to perform Live Streaming.
The Puredata Patch will use the element "MP3cast~" from the Library "Unauthorized"
to connect audio of the Computer to the Listening Port of Icecast2.

#3 the project is divided in 3 Blocs.

Bloc #1 capture mouse event and record it
Bloc #2 interprete recorded event to produce audio content
Bloc #3 Stream it Live Baby!

-----------------
| BLOC #1       |
-----------------

At the very beginning i was thinking about « XDOTOOL » to manage this part, with a little bash script and « voilà! »… But, after many try I came to the conclusion that this method slow down all the computer’s running process, because the script have to look at « click » events and record these in the same time, hum, not so easy to do… but keyloggers exist so there will be a way to do it, maybe a sneeky one.

The Python way:
Let me introduce you « pynput » a library that allows you to control and monitor input devices.

Here is the script used:

#import the library for listening inputs and date/time
from pynput.mouse import Listener
import datetime

#function used when  a « click » is detected 
#1 check the date/time && get the mouse position (x y)
#make a string with theses informations
#record it in a text file
def is_clicked(x, y, button, pressed):
	if pressed:
		d=datetime.datetime.now()
		dd=d.strftime("%d %m %Y %H %M %S")
		finalString=format(dd)+" "+format(x)+" "+format(y)+"\n"
		theFile=open("/home/pauline/theMouseRecord.txt","a", encoding="utf-8")
		
		theFile.write(finalString)
		theFile.close()
		
		
#a simple listener to trigger the clicked mouse 		
with Listener(on_click=is_clicked) as listener:
	listener.join()

Now we just have to launch the script at startup and it will be good :)

Ok, stage completed ✅ 


-----------------
| BLOC #2       |
-----------------

To make some noise, we need to use a versatile programmation language: PudeData.
I will let Hugo explain more about the sound research and work, and for now i just want to explain the very basis of the patch. We use PureData vanilla and some useful library:
"ggee" to get the "shell" object (ability to send bash commands directly from PureData)
"Unauthorized" to get the "MP3cast~" object (ability to stream audio via icecast2 from PureData)
"Zexy" to have access to a nice collections oaf objects (very useful for Hugo)


We need to read a text in PureData line by line and extract informations (date/time/x/y), so I used to launch a simple bash script from PureData using "shell" object.

The bash script:

#!/bin/bash

input="path/to/the/file.txt"

while IFS= read -r line
do
  echo "$line"
  sleep 1s
done < "$input"

Once launched the script will read and output the txt line by line and at the end the "shell" object will notice us with a "bang" like this we can relaunch the bash script when the PureData process is done.

 

Back to top