Sound System

GSoC 2011 > Sound System

In this page are described the features that will be applied to the sound related part of Planeshift. Notes about the features' implementation will be available as well.

Sound improvements

Make the Sound Manager a plugin

---

The Sound Manager today is inside the main planeshift client sources. It will be nice to have it made as a plugin to have a cleaner abstraction layer. The idea is to allow developers to use the sound system's functionalities entirely through an API. The programmers should not need to know any internal mechanics and they should never make use of internal classes of the sound system or of the Crystal Space sound classes.

The API

The module provides two interfaces: iSoundManager and iSoundControl defined as follows. These will be basically the corresponding classes of psSoundManager and SoundControl and they will have more or less similar methods. (Note that everything here is in progress: these interfaces (expecially iSoundManager) could vary a lot while I will work on the other features and improvements.)

/** This interface defines a sound manager.
*
* The sound manager controls all the sound related aspects. It manages the
* background music and the ambient sounds and it handles the music changes
* when crossing between sectors or entrying in combat mode for example.
*
* The sound manager can be used to play a sound.
*/ 
struct iSoundManager: public virtual iBase
 {
	SCF_INTERFACE(iSoundManager, 1, 1, 0);

	/**
	* The sound manager initializes by default the SoundControls with the IDs
	* in this enum. AMBIENT_SNDCTRL and MUSIC_SNDCTRL have type AMBIENT and
	* MUSIC respectively.
	*/
	enum SndCtrl_ID
	{
		AMBIENT_SNDCTRL = 0,
		MUSIC_SNDCTRL = 1,
		VOICE_SNDCTRL = 2,
		ACTION_SNDCTRL = 3,
		EFFECT_SNDCTRL = 4,
		GUI_SNDCTRL = 5
	};

	/**
	* The sound manager initialize by default the queues with the IDs specified
	* in this enum.
	*/
	enum Queue_ID
	{
		VOICE_QUEUE = 0
	};

	/**
	* In this enum are listed the possible state of a player.
	*/
	enum Combat_Stance
	{
		PEACE = 0,
		COMBAT = 1,
		DEAD = 2
	}

	//------------------//
	// SECTORS MANAGING //
	//------------------//

	/**
	* Load the sectors information into memory. If this method is not called
	* the other sector related methods have no effects. If one does not need
	* the sectors' features this method is not necessary.
	*
	* @return true if the sectors are loaded correctly, false otherwise.
	*/
	virtual bool InitializeSectors()=0;

	/**
	* Set the sector indicated as the current one. It is neither necessary nor
	* suggested to unload the previous active sector with UnloadActiveSector()
	* because the transition would result less smooth.
	*
	* The method uses the information about the sector to manage the music and
	* the ambient's sounds. If the sector has not been loaded properly with
	* InitializeSectors(), the method does nothing.
	*
	* @param sector the sector's name.
	*/
	virtual void LoadActiveSector(const char* sector)=0;

	/**
	* Unload the current active sector. The method stops both the music and all
	* the ambient's sounds.
	*/
	virtual void UnloadActiveSector()=0;

	/**
	* Reload all the sectors information in the memory.
	*/
	virtual void ReloadSectors()=0;

	//-------------------------//
	// SOUND CONTROLS MANAGING //
	//-------------------------//

	/**
	* Create and return a new SoundControl with the indicated ID and type. If a
	* SoundControl with the same ID already exists nothing happens. To use a
	* SoundControl with the same ID of an existing one, it must be first removed
	* with RemoveSndCtrl(iSoundControl* sndCtrl).
	*
	* Only one SoundControl of type AMBIENT or MUSIC can exist at the same time.
	* If an ambient(music) sound control already exists and the user tries to
	* create a new SoundControl with the same type, the previous ambient(music)
	* SoundControl's type becomes NORMAL.
	*
	* @param ctrlID the new SoundControl's identifier.
	* @param type the new SoundControl's type.
	* @return a pointer to the new SoundControl or a null pointer if a sound
	* controller with the same ID already exists.
	*/
	virtual iSoundControl* AddSndCtrl(int ctrlID, int type)=0;

	/**
	* Remove a SoundControl.
	* @param sndCtrl the SoundControl to be removed.
	*/
	virtual void RemoveSndCtrl(iSoundControl* sndCtrl)=0;

	/**
	* Get the SoundControl with the indicated ID.
	* @param ctrlID the SoundControl's ID.
	* @return the sound controller with that ID.
	*/
	virtual iSoundControl* GetSndCtrl(int ctrlID)=0;

	/**
	* Get the main SoundControl that affects the overall volume and the sound's
	* general state.
	* @return the main SoundControl.
	*/
	virtual iSoundControl* GetMainSndCtrl()=0;

	//-----------------//
	// QUEUES MANAGING //
	//-----------------//

	/**
	* Create a new sound queue with the indicated ID. The sounds of the queue
	* are played with first-in-first-out order and they are controlled by the
	* specified SoundControl.
	*
	* If a queue with the same ID already exists nothing happens. To create a
	* new queue with its same ID one has to remove the old one with the method
	* RemoveSndQueue(int queueID).
	*
	* @param queueID the queue's ID.
	* @param sndCtrl the SoundControl of the queue's sounds.
	* @return true if the queue is created, false if another one with the same
	* ID already exists.
	*/
	virtual bool AddSndQueue(int queueID, iSoundControl* sndCtrl)=0;

	/**
	* Remove the queue with the indicated ID.
	* @param queueID the queue's ID.
	*/
	virtual void RemoveSndQueue(int queueID)=0;

	/**
	* Push a new sound in the queue with the indicated ID. The sound is played
	* when all the item pushed before it have been played.
	* 
	* @param queueID the queue's ID.
	* @param fileName the file's name of the sound to play.
	* @return true if a queue with that ID exists, false otherwise.
	*/
	virtual bool PushQueueItem(int queueID, const char* fileName)=0;

	//----------------//
	// STATE MANAGING //
	//----------------//

	/**
	* Set the new combat stance and starts the combat music if the combat toggle
	* is on. The new value should be one of the enum Combat_Stance.
	* @param newCombatStance the new combat state of the player.
	*/
	virtual void SetCombatStance(int newCombatStance)=0;

	/**
	* Get the current combat stance.
	* @return the combat stance.
	*/
	virtual int GetCombatStance() const=0;

	/**
	* Set the player's position.
	* @param playerPosition the player's position.
	*/
	virtual void SetPosition(csVector3 playerPosition)=0;

	/**
	* Get the player's position.
	* @return the player's position.
	*/
	virtual csVector3 GetPosition() const=0;

	/**
	* Set the time of the day. The method works only if sectors have been
	* initialized.
	* @param newTimeOfDay the new time of the day.
	*/
	virtual void SetTimeOfDay(int newTimeOfDay)=0;

	/**
	* Get the current time of the day. The method works only if sectors have
	* been initialized.
	* @return the time of the day or -1 if the sectors are not initialized.
	*/
	virtual int GetTimeOfDay() const=0;

	/**
	* Set the weather's state.
	* @param newWeather the weather to be set.
	*/
	virtual void SetWeather(int newWeather)=0;

	/**
	* Get the weather's state.
	* @return the weather's state.
	*/
	virtual int GetWeather() const=0;

	/**
	* Sets the new state for the entity associated to the given mesh and
	* plays the start resource (if defined). If it is already playing a
	* sound, it is stopped.
	*
	* @param state the new state > 0 for the entity. For negative value
	* the function is not defined.
	* @param mesh the mesh associated to the entity.
	* @param forceChange if it is false the entity does not change its
	* state if the new one is not defined. If it is true the entity stops
	* play any sound until a new valid state is defined.
	*/
	virtual void SetEntityState(int state, iMeshWrapper* mesh, bool forceChange) = 0;

	//------------------//
	// TOGGLES MANAGING //
	//------------------//

	/**
	* Set the value of background music toggle. If set to true the background
	* music loops, otherwise it does not.
	* @param toggle true to activate the background music loop, false otherwise.
	*/
	virtual void SetLoopBGMToggle(bool toggle)=0;

	/**
	* Get the value of LoopBGMToggle.
	* @return true if the background music loop is activated, false otherwise.
	*/
	virtual bool IsLoopBGMToggleOn()=0;

	/**
	* Set the value of the combat music toggle. If set to true a change in the
	* combat stance changes the sector's music accordingly.
	* @param toggle true to allow the music change, false otherwise.
	*/
	virtual void SetCombatMusicToggle(bool toggle)=0;

	/**
	* Get the value of the combat music toggle.
	* @return true if the music change with the combat stance, false otherwise.
	*/
	virtual bool IsCombatMusicToggleOn()=0;

	/**
	* Set the value of the listener on camera toggle. If set to true the
	* listener takes the camera's position, otherwise the player's one.
	* @param toggle true to place the listener on the camera, false otherwise.
	*/
	virtual void SetListenerOnCameraToggle(bool toggle)=0;

	/**
	* Get the value of the listener on camera toggle.
	* @return true if the listener is placed on the camera, false otherwise.
	*/
	virtual bool IsListenerOnCameraToggleOn()=0;

	/**
	* Set the value of the chat toggle needed to keep track of the chat's sound state.
	*/
	virtual void SetChatToggle(bool toggle)=0;

	/**
	* Get the chat toggle value.
	*/
	virtual bool IsChatToggleOn()=0;

	//------------//
	// PLAY SOUND //
	//------------//

	/**
	* Play a 2D sound.
	* @param fileName the name of the file where the sound is stored.
	* @param loop true if the sound have to loop, false otherwise.
	* @param ctrl the SoundControl that handle the sound.
	*/
	virtual void PlaySound(const char* fileName, bool loop, iSoundControl* &ctrl)=0;

	/**
	* Play a 3D sound.
	* @param fileName the name of the file where the sound is stored.
	* @param loop true if the sound have to loop, false otherwise.
	* @param ctrl the SoundControl that handle the sound.
	* @param pos the position of the sound source.
	* @param dir the direction of the sound.
	* @param minDist the minimum distance at which the player can hear it.
	* @param maxDist the maximum distance at which the player can hear it.
	*/
	virtual void PlaySound(const char* fileName, bool loop, iSoundControl* &ctrl, csVector3 pos, csVector3 dir, float minDist, float maxDist)=0;

	/**
	* Stop a sound with the indicated name.
	* @param fileName the name of the file where the sound is stored.
	* @return true if a sound with that name exists, false otherwise.
	*/
	virtual bool StopSound(const char* fileName)=0;

	/**
	* Set the sound source position.
	* @param fileName the name of the file where the sound is stored.
	* @param position the new position of the sound source.
	* @return true if a sound with that name exists, false otherwise.
	*/
	virtual bool SetSoundSource(const char* fileName, csVector3 position)=0;

	//--------//
	// UPDATE //
	//--------//

	/**
	* Update the sound manager. Update all non event based things.
	*/
	virtual void Update()=0;

	/**
	* Update the position of the listener. If the listener on camera toggle
	* is on, the listener's position is set on the camera otherwise on the
	* player's position.
	*
	* @param the view of the camera.
	*/
	virtual void UpdateListener(iView* view)=0;
};

/** This interface defines the sound controller used by the application.
*
* The API allows the user to control the volume and the state of a group
* of sounds. The user can define a different SoundControl for each type
* of sound is needed to be controlled differently. For example one can
* use a SoundControl to manage voices and another one for the GUI's sounds.
*/
struct iSoundControl
{
	/**
	* Every SoundControl has a type as defined in this enum. There are two
	* special types: AMBIENT and MUSIC that control the ambient and the music
	* sounds respectively. There can be only one SoundControl of these two
	* types at the same time while there can be an unlimited number of sound
	* controllers with type NORMAL.
	*/
	enum SndCtrl_Type
	{
		NORMAL = 0,
		AMBIENT = 1,
		MUSIC = 2
	};

	/**
	* Get the SoundControl's ID.
	* @return the Soundcontrol's ID.
	*/
	virtual int GetID() const=0;

	/**
	* Get the SoundControl's type.
	* @return the SoundControl's type.
	*/
	virtual int GetType() const=0;

	/**
	* Get the current volume of the sounds controlled by this SoundControl.
	* @return the volume as a float.
	*/
	virtual float GetVolume() const=0;

	/**
	* Set the volume of the sounds controlled by this SoundControl.
	* @param vol the volume to be set.
	*/
	virtual void SetVolume(float vol)=0;

	/**
	* Unmute sounds controlled by this SoundControl.
	*/
	virtual void Unmute()=0;

	/**
	* Mute sounds controlled by this SoundControl.
	*/
	virtual void Mute()=0;

	/**
	* Get the current Toggle state.
	* @return true if sounds controlled by this SoundControl are activated,
	* false otherwise.
	*/
	virtual bool GetToggle() const=0;

	/**
	* Set the current Toggle state.
	* @param toggle true to activate the sounds controlled by this SoundControl,
	* false otherwise.
	*/
	virtual void SetToggle(bool toggle)=0;

	/**
	* Deactivate sounds controlled by this SoundControl.
	*/
	virtual void DeactivateToggle()=0;

	/**
	* Activate sounds controlled by this SoundControl.
	*/
	virtual void ActivateToggle()=0;
};

Comments and differences with psSoundManager and SoundControl

• I still do not like the way PlaySound are handled here. I will find a better solution later.
• I have changed the name of Load(), Unload() and Reload() to the more explanatory LoadActiveSector(), UnloadActiveSector() and ReloadSectors().
• Note that the main SoundControl is treated differently as it controls the overall properties.
• The interface lets the user to handle more than a queue of sounds. Right now psSoundManager uses only one queue for NPCs' voices.
• Methods similar to psSoundManager::SetVoiceToggle() and psSoundManager::GetVoiceToggle() are not available since the user can access directly to the voice sound control through GetSndCtrl() (actually those methods are not needed even in psSoundManager).
• psSoundManager uses 4 psToggle. That class add to a boolean toggle only the callback feature that should not be accessed by external classes so I have decided to present an interface with only getters and setters. psToggle will still be used in the plugin module's implementation.
• Note that iSoundControl does not support the callback functionalities; indeed I do not think the user should use them. Those methods will be anyway implemented in the plugin module for internal uses.
• The only add to iSOundControl is the type.

Sounds in factories

---

In CS every object has a factory and multiple instances, so you can have a mesh factory, which is a rock, and then instanciate it multiple times by changing its coords, rotation and texture. It is possible now to associate a sound to factories and meshes; in this way it is easy to make a monster play sounds for example.

The ENTITY tag

A sound can be associated to a mesh factory or a mesh by using the tag <ENTITY /> in an xml document that describe an area (currently these documents are located in /art/soundlib.zip/areas). There are two types of entities: a factory entity and a mesh entity. The first one associates a sound to a factory, the second one to a mesh. Here there are two example that explain how to create an entity:

<ENTITY FACTORY="name_of_the_factory" MAX_RANGE="10.5">
	<STATE ID="1">
		<RESOURCE NAME="name_of_the_sound_to_play" />
	</STATE>
</ENTITY>

<ENTITY MESH="name_of_the_mesh" MAX_RANGE="10.5">
	<STATE ID="1">
		<RESOURCE NAME="name_of_the_sound_to_play" />
	</STATE>
</ENTITY>

In these two examples there are all the mandatory attributes and elements of the ENTITY TAG:

• One and only one attribute between FACTORY and MESH that contains the name of the factory/mesh that you want to play the sound. If FACTORY is specified the entity is considered a factory entity, if MESH is given it is considered a mesh entity. If both those attributes are given, only the attribute MESH is considered. If no one of them is specified the tag is just ignored.
• One attribute MAX_RANGE that specifies the maximum distance at which the sound can be heard.
• At least one child tag STATE with one attribute ID. ID must be non-negative. The state is used to determine in which situation the factory (mesh) will play the sound. For example a factory called "rat" can be associated to a calm peep when it is in a normal state and to a more aggressive noise when it enters in combat. There can be an unlimited number of state for each entity. The states used by the client are those in the enum psModeMessage::playerMode.
• At least one child tag RESOURCE with one attribute NAME. NAME gives the name of the sound that must be played randomly during the period of time in which the factory (mesh) is in the given state. There can be an unlimited number of resource for each state; when the entity plays a sound, it picks randomly one of the given resources.

Optional attributes

There are other optional attributes that can be used in the tag to configure better an entity. Below there is a complete example with all the attributes that can be used and their explanation.

<ENTITY FACTORY="name_of_the_factory" MIN_RANGE="0.5" MAX_RANGE="10.5">
	<STATE ID="1" PROBABILITY="0.5" DELAY="20" VOLUME="1.0" TIME_START="9" TIME_END="21" FALLBACK_STATE="2" FALLBACK_PROBABILITY="0.5">
		<RESOURCE NAME="name_of_the_sound_to_play_1" />
		<RESOURCE NAME="name_of_the_sound_to_play_2" />
		<RESOURCE NAME="name_of_the_sound_to_play_3" />
	</STATE>

	<STATE ID="2" PROBABILITY="1.0" DELAY="20" VOLUME="1.0" TIME_START="0" TIME_END="24">
		<RESOURCE NAME="name_of_the_sound_to_play_1" />
	</STATE>
</ENTITY>

• MIN_RANGE: the maximum distance at which the sound is heard with the maximum volume.
• PROBABILITY: the probability that one of the resources is played in a second.
• DELAY: the interval of time in seconds during which the factory (mesh) cannot play any sound once the previous sound is finished.
• VOLUME: the general volume of the sound (it will decrease with the distance).
• TIME_START: the time of the day in hours when the factory (mesh) can play the sound when it's in that state.
• TIME_END: the time of the day in hours when the factory (mesh) can't play the sound when it's in that state.
• FALLBACK_STATE: the ID of the state that is activated after this one has been activated with a probability per second given by FALLBACK_PROBABILITY.
• FALLBACK_PROBABILITY: the probability per second that the entity state changes into the one give by FALLBACK_STATE.

The optional attributes, if not given, assume their default values that are:

• MIN_RANGE = 0.0;
• PROBABILITY = 1.0;
• DELAY = 0;
• VOLUME = 1.0;
• TIME_START = 0;
• TIME_END = 24;
• FALLBACK_STATE = UNDEFINED_STATE;
• FALLBACK_PROBABILITY = 0.0;

Example

We want to make a monster emits different sounds when it is in different states. The states are peace, combat and died. Moreover we want the monster to emit a sound when it charges the player:

<ENTITY FACTORY="monster_factory" MAX_RANGE="30.0">
 	<!-- Peace state -->
 	<STATE ID="1" PROBABILITY="0.1" DELAY="10">
 		<RESOURCE NAME="peace_sound_1" />
 		<RESOURCE NAME="peace_sound_2" />
 		<RESOURCE NAME="peace_sound_3" />
 	</STATE>
 
 	<!-- Charge -->
 	<STATE ID="2" PROBABILITY="1.0" DELAY="10" VOLUME="1.5" FALLBACK_STATE="100" FALLBACK_PROBABILITY="1.0">
 		<RESOURCE NAME="charge_sound_1" />
 		<RESOURCE NAME="charge_sound_2" />
 		<RESOURCE NAME="charge_sound_3" />
 	</STATE>
 
 	<!-- Dead state -->
 	<STATE ID="5" PROBABILITY="1.0" FALLBACK_PROBABILITY="1.0">
 		<RESOURCE NAME="dead_sound_1" />
 		<RESOURCE NAME="dead_sound_2" />
 		<RESOURCE NAME="dead_sound_3" />
 	</STATE>

 	<!-- Combat state -->
 	<STATE ID="100" PROBABILITY="0.5" DELAY="3">
 		<RESOURCE NAME="combat_sound_1" />
 		<RESOURCE NAME="combat_sound_2" />
 		<RESOURCE NAME="combat_sound_3" />
 	</STATE>
 </ENTITY>

Let's examine it in details. The first state is the peace state. Its ID is 1 (the same as the PEACE mode in psModeMessage::playerMode). PROBABILITY="0.1" means that there is a probability of 0.1 that the monster will play one of the resources in a second; moreover DELAY is set to 10 seconds so (on average) the monster will play once every 20 seconds (for 10 seconds the monster is surely silent because of the delay, the remaining 10 seconds are random and associated to the probability). All the other parameters not explicitly written take the default value as explained in the section above.

The second state is associated to the charge event. The ID is 2. If we take a look at psModeMessage::playerMode we notice that this is the ID for the combat state. The probability is 1.0 so the sound is played as soon as the monster enter in combat mode. Since FALLBACK_PROBABILITY is 1.0 the monster will change its state to the one with ID="100" right after the sound has been played. The state with ID="100" is defined at the end and its the state associated with the combat state. By using the fallback state we can make the monster emit a sound just when it enters in combat mode (in this case a shout at the beginning of the combat) and then change the set of resources that are played (in this case to the resources in the state 100, the fight sounds).

The third state is the dead state. As you can see the FALLBACK_PROBABILITY is set to 1.0 but there is not the FALLBACK_STATE! In this way the entity will change its state to UNDEFINED_STATE (see default values above) and won't play anything else. The monster thus will play a sound as soon as it will be killed and then it will change its state and it won't play anything else.

The common sector

It is possible to create an xml document that define a sector called "common". The sound manager plugin treats this sector differently. You can use the common sector to describe the properties common to all sectors. At the current time the sound plugin manager supports only the entities of the common sectors. BACKGROUND, EMITTER and AMBIENT tags are just ignored.

Ambiguities and priorities

When a state with the same ID is defined more than once for the same factory (mesh) entity, only the first one is picked up. All the other ones are ignored.

In summary the sound for a mesh can be specified in four ways: a mesh entity in its sector, a factory entity in its sector (that defines the sound also for all the others meshes produced by its factory), a mesh entity in the common sector and a factory entity in the common sector. These four ways have a different priority: the mesh entities in the current sector have the maximum priority and the factory entities in the common sector the minimum one. The complete priority order order is the following:

1. mesh entity in its sector;
2. factory entity in its sector;
3. mesh entity in the common sector;
4. factory entity in the common sector.

Distance lag effect of 3d emitters

---

Sound has a certain speed and openal unfortunately doesn't handle this factor so, even if it lowers the volume of distant sources, it will still make the sound of them run at the same time making the actual volume sound higher than it is actually. This would be about making emitters slow down when the player goes away from the source in order to unsync it from the other, more near emitters and go back to the normal speed after it has reached a certain offset from the more near object, in order to simulate this effect of distance and avoid equal sounds to stack.

Implementation notes

The sound's speed has been simulated in two different ways in the plugin:

1. 3D sounds are played after a delay that depends on the distance between the source and the player. This one leans on the CS' event timer so it is not very precise but still it can be used to make differences between near sources and far away ones. Moreover this is the only effect that takes place when the player is not moving.
2. 3D sounds are now subjected to Doppler effect. The source's speed is not taken into account to save a bit of time and memory. This means that the relative speed between an unmoving player and a moving monster is 0 m/s. This is not a big issue since this system is thought to unstack sounds and it's very unlikely that more than a moving source emits the same sound at the same time. This aspect takes care of changing the play rate speed of near sounds. It does the same for far away sounds but it cannot discriminate well between two adjacent sources far from the player.

The formulas to compute the delay and the change of frequency are:

1. delay = d / SPEED_OF_SOUND
2. heard_frequency = f0 * (SPEED_OF_SOUND - DOPPLER_FACTOR * v) / SPEED_OF_SOUND

where d is the distance between the player and the source, f0 is he emitted frequency and v is the speed of the player toward the source (positive if the player is going away from the source, negative in the other case). DOPPLER_FACTOR is just used to weight the contribution of the relative speed v.

Both the constants SPEED_OF_SOUND and DOPPLER_FACTOR are defined in the applications' configuration file. Their default value are set to 331 and 2.5 respectively. To configure the effect of the distance lag effect you can change those values.

Not-Doppler emitters

It is possible to disable the Doppler effect directly in the xml file that define them with the optional parameter DOPPLER_ENABLED. Its default value is true. To disable the Doppler effect set it to false as in the example.

<EMITTER RESOURCE="blabla" ... DOPPLER_ENABLED="false" />

Overall code cleanup

---

Here will follow a list of the cleanup interventions in the code.

Sounds events

We need to allow to play a sound when some events happen.

Random sounds on monsters

When they stand still and idle. If the race is an intelligent race, then we can play some phrases, if not we can just have screams/sounds. These sounds can be played by using the factory sound system described above.

Sounds associated to behaviours/actions of monsters

Like he is angry and plays the angry sound. These sounds can be played by using the factory sound system described above.

Random sounds from environment

Buzzes of flies, water for rivers, wind and so on so forth. These sounds can be played using emitters.

Musical instruments

This will allow players to use musical instruments.

Sound part

---

This part implements the conversion between the musical sheet and the music that it represents. There are three major problems in this part:

• the song should be created from sound samples (the notes) stored in file of any format.
• There will be likely quite large musical sheets thus converting them into music all at once could be too much time consuming and lead to slowdowns of the client. Breaking down the song into pieces and play them one at a time is quite impossible to do because the event framework offered by CS is not enough precise to allow the right timing required to play the whole song. The only other solution is to update the sound stream of the song as the musical sheet is translated.
• treat a song as a normal sound in order to be able to use all the other implemented features.

There will be four main classes: SndSysSongData, SndSysSongStream, SongHandle and Instrument. These four will be likely be managed by a InstrumentsManager.

Instrument

This class represents a musical instrument. It keeps the decoded data of all the notes that it can play and it provides this data when asked. Data of notes comes from the files that contain note samples. Instrument first converts the file into a iSndSysStream and reads the data from it. Since Instrument reads directly from the stream, the data will be already decoded thus the sample file format is not important.

All the instruments are defined in an XML document that InstrumentManager reads when created. The syntax of the XML language can be understood completely with this example:

<instruments>
	<instrument name="guitar" polyphony="3" volume="1.0" min_dist="1.0" max_dist="50.0">
		<note resource="E2" step="E" alter="0" octave="2"/>
		<note resource="A3" step="A" alter="0" octave="3"/>
		<note resource="Bb3" step="B" alter="-1" octave="3"/>
		<note resource="C#4" step="C" alter="1" octave="4"/>
		<note resource="D#4" step="D" alter="1" octave="4"/>
		<note resource="E4" step="E" alter="0" octave="4"/>
	</instrument>
</instruments>

<instruments> is the root. The tag <instrument> defines an instrument with its name, polyphony (i.e. the number of notes that it can plays at the same time), volume, the minimum distance where the sound is heard louder and the maximum distance at which it can be heard. <note> defines a note. The parameter resource indicates what's the name of the file that still need to be defined in soundlib.xml; step is the note's pitch; alter is the alteration of the note (-1 flat, 0 unaltered, 1 sharp); octave is the octave of the note (4 is the central octave in piano).

Note: there is no need to define an enharmonic not of an alredy defined one. For example, defining B flat will automatically defines also A sharp; defining E will automatically define also F flat and so on so forth.

Important: the instrument will provide the data to the sound plugin with the format of the file that the first defined note uses; thus, if notes with different formats are used, it's better to define as first the sound with the better quality. Instrument will convert the files with different formats into that one. Remember that when data is converted from a format with a very low sample frequency to one with high sample frequency, the quality can decrease a lot. If there are files with very different sample frequency maybe it's better to put as first the note that use a file with an intermediate sample frequency.

SndSysSongData and SndSysSongStream

SndSysSongData and SndSysSongStream implement respectively iSndSysData and iSndSysStream. The instrument system is interfaced with the CS' sound renderer as a stream with unknown length. The stream keeps the musical sheet and the instrument to play it with. When the renderer calls iSndSysStream::GetDataPointers(...) to fill its buffer, SndSysSongStream generates the needed data dinamically by translating the sheet and using the instrument to retrieve the data of the notes. The sheet is thus translated just as needed each time saving in this way memory and distributing the computation needed to parse the XML document through time.

SongHandle

SongHandle is children of SoundHandle. It is basically the same as its parent class but it uses SndSysSongData as sound data and SndSysSongStream as its sound stream. This handle allows the song to be played as any other sounds with SoundSystemManager::Play3DSound(...). This means that the song are able to use all the already implemented features of other sounds (for example it can be stopped as all the other sounds).

GUI part

---

Figure 1: The GUI used to create musical sheets in "edit mode"

In Figure 1 is represented the GUI to create musical sheet. The toolbar presents many buttons, in order from left to right:

1. Edit mode (toggle): toggle on-off the edit mode (where the user can edit the sheet).
2. Play/Stop (toggle): used to play/stop the musical sheet.
3. Load: load a sheet from a file on the hard disk.
4. Save: save the sheet into a file on the hard disk.
5. Change title: change the title of the song.
6. Switch double staff: toggle on-off the double staff (see Figure 2).
7. Tonality: change the tonality of the sheet.
8. Meter: change the meter of the song /3/4, 4/4, etc.)
9. BPM: change the BPM of the song.
10. Rest (toggle): when down rests are inserted instead of notes.
11. The third group of buttons are the duration buttons and they decide the duration of the note/rest. They are all toggle buttons. In order they are: sixteenth, eighth, quarter, half and whole. The last one is a toggle button too and when down a dot is inserted near the note to multiply its duration by 1.5 times.
12. The fourth group of buttons are the alteration buttons, in order flat, natural and sharp. They are all toggle buttons.
13. Delete chord: delete the selected chord.
14. Delete measure: delete the selected measure.

Figure 2: The GUI used to create musical sheets with double staff

1. Insert note (toggle): insert a note where clicked.
2. Insert measure: insert a note before the selected measure.
3. Start repeat (toggle): the selected measure becomes the start of a repeated part.
4. End repeat (toggle): the selected measure becomes the end of a repeated part.
5. Ending (toggle): the selected measure becomes an ending.

Toggle buttons need 2 images (the up button image and the down one).

Suggested images

For toggle buttons (with the exception of the Play/Stop button) it would be nice to have a shaded background version of the button when it is down.

• Play/Stop (toggle): a play symbol when the button is off, a stop symbol when the button is on.
• Load: I think that the given image is enough, it's the same that the sketch interface so the user is already acquainted with it.
• Save: I think that the given image is enough, it's the same that the sketch interface so the user is already acquainted with it.
• Change title: I think that the given image is enough, it's the same that the sketch interface so the user is already acquainted with it.
• Tonality: something like this. You can avoid to draw the clef.
• Switch double staff: a staff.
• Meter: a fraction 4/4 like here
• BPM: just write "BPM".
• Rest (toggle): a quarter rest (e.g. the third symbol in this image.
• Duration buttons: a note with the corresponding duration, take a look here to find the symbol associate to each note value.
• Dot button: a dot.
• Alteration buttons: a flat for the first button, a natural for the second one and a sharp for the third one.
• Delete chord: a note with a cross on it.
• Delete measure: a measure with a cross on it.
• Start repeat (toggle): a line with a colon on its right.
• End repeat (toggle): a line with a colon on its left like here.
• Ending (toggle): something like this (the upper line doesn't have to be so long).

Network communications part

---

When a player click the Play Song button, the song manager send the compressed musical sheet to the server asking for playing it. The server then does the following things:

• checks that the player is the owner of the musical sheet;
• checks that the player has an instrument equipped;
• determines with which instrument the player should play the song;
• computes the error probabilities depending on the player skills with that instrument;
• send a psPlaySongMessage containing the score, the instrument to use and the error probabilities to the player and all the near players.

The client receive all these parameters and translate the score into music.

Note that the server can only compute the error probabilities but it's the client that applies it. This is necessary because the server doesn't control directly the produced sound but it has only the musical sheet in an XML format. Another approach to this problem would be letting the server change the played musical sheet depending on the error probabilities. This would still be uneffective because:

• the score is saved on the server compressed so to be edited it should be decompressed first (time consuming).
• the score should be parsed and edited (very time consuming).
• the client would still be hackable by changing the received musical sheet with the one represented in pawsMusicWindow that is error-free.

Thus, any attempt to let the server decide how to apply the error probability to the song's execution is useless. Nevertheless this is not a big deal:

• the computed error probability affects only what the player hear but not the sucess/fail of the operation (that is handled by the server).
• even if the player hacked his client to set the error probability to 0, all the near players would hear that song with mistakes.

Optional extensions

These things (or part of them) will be done during the GSoC coding time only time permitting. Otherwise they can still be included in the next GSoC or implemented by developers.

• Sounds for each weapon and attack: if you attack with a sword you should hear the sound of a sword.
• Add ability to limit number of channels. The idea is to be able to limit the amount of sounds played at the same time coming from different sources. In the same areas you can have rivers, monsters, wind, and even the player clicking on the UI.
• Special actions to do when particular sound types are being played (for example reducing all other sounds/music volume)
• Allow voice speech in game through peer to peer connection.
• Have the walking sounds as a possible perception for monsters nearby. like a thief moving close to a monster to attack him first without making too much noise by walking, while a player with no sneak skills, will just make more noise while walking and the monster will hear this and attach him first.
• Have the sound of a player walking that vary based on the ground; so if it's rocky texture it's harder sound, and if it's grass is softer sound.

Skills needed

Our server and client are all coded in C++ and the sounds are defined in XML files. For this project C++ knowledge is needed, some basic knowledge of XML. No MySQL knowledge is required.

Difficulty

medium to difficult, depending on the parts of this which are picked up