Sound System
GSoC 2011 > Sound System
This page is a WORK IN PROGRESS edited by Lucubro. Information in this page are not definitive. If you have comments you can leave a message in the discussion page or in my personal talk.
In this page are described the features that will be applied to the sound related part of Planeshift. Notes about the features' implementation will be available as well.
Sound improvements
Make the Sound Manager a plugin
The Sound Manager today is inside the main planeshift client sources. It will be nice to have it made as a plugin to have a cleaner abstraction layer. The idea is to allow developers to use the sound system's functionalities entirely through an API. The programmers should not need to know any internal mechanics and they should never make use of internal classes of the sound system or of the Crystal Space sound classes.
The API
The module provides two interfaces: iSoundManager and iSoundControl defined as follows. These will be basically the corresponding classes of psSoundManager and SoundControl and they will have more or less similar methods. (Note that everything here is in progress: these interfaces (expecially iSoundManager) could vary a lot while I will work on the other features and improvements.)
/** This interface defines a sound manager. * * The sound manager controls all the sound related aspects. It manages the * background music and the ambient sounds and it handles the music changes * when crossing between sectors or entrying in combat mode for example. * * The sound manager can be used to play a sound. */ struct iSoundManager: public virtual iBase { SCF_INTERFACE(iSoundManager, 1, 1, 0); /** * The sound manager initializes by default the SoundControls with the IDs * in this enum. AMBIENT_SNDCTRL and MUSIC_SNDCTRL have type AMBIENT and * MUSIC respectively. */ enum SndCtrl_ID { AMBIENT_SNDCTRL = 0, MUSIC_SNDCTRL = 1, VOICE_SNDCTRL = 2, ACTION_SNDCTRL = 3, EFFECT_SNDCTRL = 4, GUI_SNDCTRL = 5 }; /** * The sound manager initialize by default the queues with the IDs specified * in this enum. */ enum Queue_ID { VOICE_QUEUE = 0 }; /** * In this enum are listed the possible state of a player. */ enum Combat_Stance { PEACE = 0, COMBAT = 1, DEAD = 2 } //------------------// // SECTORS MANAGING // //------------------// /** * Load the sectors information into memory. If this method is not called * the other sector related methods have no effects. If one does not need * the sectors' features this method is not necessary. * * @return true if the sectors are loaded correctly, false otherwise. */ virtual bool InitializeSectors()=0; /** * Set the sector indicated as the current one. It is neither necessary nor * suggested to unload the previous active sector with UnloadActiveSector() * because the transition would result less smooth. * * The method uses the information about the sector to manage the music and * the ambient's sounds. If the sector has not been loaded properly with * InitializeSectors(), the method does nothing. * * @param sector the sector's name. */ virtual void LoadActiveSector(const char* sector)=0; /** * Unload the current active sector. The method stops both the music and all * the ambient's sounds. */ virtual void UnloadActiveSector()=0; /** * Reload all the sectors information in the memory. */ virtual void ReloadSectors()=0; //-------------------------// // SOUND CONTROLS MANAGING // //-------------------------// /** * Create and return a new SoundControl with the indicated ID and type. If a * SoundControl with the same ID already exists nothing happens. To use a * SoundControl with the same ID of an existing one, it must be first removed * with RemoveSndCtrl(iSoundControl* sndCtrl). * * Only one SoundControl of type AMBIENT or MUSIC can exist at the same time. * If an ambient(music) sound control already exists and the user tries to * create a new SoundControl with the same type, the previous ambient(music) * SoundControl's type becomes NORMAL. * * @param ctrlID the new SoundControl's identifier. * @param type the new SoundControl's type. * @return a pointer to the new SoundControl or a null pointer if a sound * controller with the same ID already exists. */ virtual iSoundControl* AddSndCtrl(int ctrlID, int type)=0; /** * Remove a SoundControl. * @param sndCtrl the SoundControl to be removed. */ virtual void RemoveSndCtrl(iSoundControl* sndCtrl)=0; /** * Get the SoundControl with the indicated ID. * @param ctrlID the SoundControl's ID. * @return the sound controller with that ID. */ virtual iSoundControl* GetSndCtrl(int ctrlID)=0; /** * Get the main SoundControl that affects the overall volume and the sound's * general state. * @return the main SoundControl. */ virtual iSoundControl* GetMainSndCtrl()=0; //-----------------// // QUEUES MANAGING // //-----------------// /** * Create a new sound queue with the indicated ID. The sounds of the queue * are played with first-in-first-out order and they are controlled by the * specified SoundControl. * * If a queue with the same ID already exists nothing happens. To create a * new queue with its same ID one has to remove the old one with the method * RemoveSndQueue(int queueID). * * @param queueID the queue's ID. * @param sndCtrl the SoundControl of the queue's sounds. * @return true if the queue is created, false if another one with the same * ID already exists. */ virtual bool AddSndQueue(int queueID, iSoundControl* sndCtrl)=0; /** * Remove the queue with the indicated ID. * @param queueID the queue's ID. */ virtual void RemoveSndQueue(int queueID)=0; /** * Push a new sound in the queue with the indicated ID. The sound is played * when all the item pushed before it have been played. * * @param queueID the queue's ID. * @param fileName the file's name of the sound to play. * @return true if a queue with that ID exists, false otherwise. */ virtual bool PushQueueItem(int queueID, const char* fileName)=0; //----------------// // STATE MANAGING // //----------------// /** * Set the new combat stance and starts the combat music if the combat toggle * is on. The new value should be one of the enum Combat_Stance. * @param newCombatStance the new combat state of the player. */ virtual void SetCombatStance(int newCombatStance)=0; /** * Get the current combat stance. * @return the combat stance. */ virtual int GetCombatStance() const=0; /** * Set the player's position. * @param playerPosition the player's position. */ virtual void SetPosition(csVector3 playerPosition)=0; /** * Get the player's position. * @return the player's position. */ virtual csVector3 GetPosition() const=0; /** * Set the time of the day. The method works only if sectors have been * initialized. * @param newTimeOfDay the new time of the day. */ virtual void SetTimeOfDay(int newTimeOfDay)=0; /** * Get the current time of the day. The method works only if sectors have * been initialized. * @return the time of the day or -1 if the sectors are not initialized. */ virtual int GetTimeOfDay() const=0; /** * Set the weather's state. * @param newWeather the weather to be set. */ virtual void SetWeather(int newWeather)=0; /** * Get the weather's state. * @return the weather's state. */ virtual int GetWeather() const=0; /** * Sets the new state for the entity associated to the given mesh and * plays the start resource (if defined). If it is already playing a * sound, it is stopped. * * @param state the new state > 0 for the entity. For negative value * the function is not defined. * @param mesh the mesh associated to the entity. * @param forceChange if it is false the entity does not change its * state if the new one is not defined. If it is true the entity stops * play any sound until a new valid state is defined. */ virtual void SetEntityState(int state, iMeshWrapper* mesh, bool forceChange) = 0; //------------------// // TOGGLES MANAGING // //------------------// /** * Set the value of background music toggle. If set to true the background * music loops, otherwise it does not. * @param toggle true to activate the background music loop, false otherwise. */ virtual void SetLoopBGMToggle(bool toggle)=0; /** * Get the value of LoopBGMToggle. * @return true if the background music loop is activated, false otherwise. */ virtual bool IsLoopBGMToggleOn()=0; /** * Set the value of the combat music toggle. If set to true a change in the * combat stance changes the sector's music accordingly. * @param toggle true to allow the music change, false otherwise. */ virtual void SetCombatMusicToggle(bool toggle)=0; /** * Get the value of the combat music toggle. * @return true if the music change with the combat stance, false otherwise. */ virtual bool IsCombatMusicToggleOn()=0; /** * Set the value of the listener on camera toggle. If set to true the * listener takes the camera's position, otherwise the player's one. * @param toggle true to place the listener on the camera, false otherwise. */ virtual void SetListenerOnCameraToggle(bool toggle)=0; /** * Get the value of the listener on camera toggle. * @return true if the listener is placed on the camera, false otherwise. */ virtual bool IsListenerOnCameraToggleOn()=0; /** * Set the value of the chat toggle needed to keep track of the chat's sound state. */ virtual void SetChatToggle(bool toggle)=0; /** * Get the chat toggle value. */ virtual bool IsChatToggleOn()=0; //------------// // PLAY SOUND // //------------// /** * Play a 2D sound. * @param fileName the name of the file where the sound is stored. * @param loop true if the sound have to loop, false otherwise. * @param ctrl the SoundControl that handle the sound. */ virtual void PlaySound(const char* fileName, bool loop, iSoundControl* &ctrl)=0; /** * Play a 3D sound. * @param fileName the name of the file where the sound is stored. * @param loop true if the sound have to loop, false otherwise. * @param ctrl the SoundControl that handle the sound. * @param pos the position of the sound source. * @param dir the direction of the sound. * @param minDist the minimum distance at which the player can hear it. * @param maxDist the maximum distance at which the player can hear it. */ virtual void PlaySound(const char* fileName, bool loop, iSoundControl* &ctrl, csVector3 pos, csVector3 dir, float minDist, float maxDist)=0; /** * Stop a sound with the indicated name. * @param fileName the name of the file where the sound is stored. * @return true if a sound with that name exists, false otherwise. */ virtual bool StopSound(const char* fileName)=0; /** * Set the sound source position. * @param fileName the name of the file where the sound is stored. * @param position the new position of the sound source. * @return true if a sound with that name exists, false otherwise. */ virtual bool SetSoundSource(const char* fileName, csVector3 position)=0; //--------// // UPDATE // //--------// /** * Update the sound manager. Update all non event based things. */ virtual void Update()=0; /** * Update the position of the listener. If the listener on camera toggle * is on, the listener's position is set on the camera otherwise on the * player's position. * * @param the view of the camera. */ virtual void UpdateListener(iView* view)=0; };
/** This interface defines the sound controller used by the application. * * The API allows the user to control the volume and the state of a group * of sounds. The user can define a different SoundControl for each type * of sound is needed to be controlled differently. For example one can * use a SoundControl to manage voices and another one for the GUI's sounds. */ struct iSoundControl { /** * Every SoundControl has a type as defined in this enum. There are two * special types: AMBIENT and MUSIC that control the ambient and the music * sounds respectively. There can be only one SoundControl of these two * types at the same time while there can be an unlimited number of sound * controllers with type NORMAL. */ enum SndCtrl_Type { NORMAL = 0, AMBIENT = 1, MUSIC = 2 }; /** * Get the SoundControl's ID. * @return the Soundcontrol's ID. */ virtual int GetID() const=0; /** * Get the SoundControl's type. * @return the SoundControl's type. */ virtual int GetType() const=0; /** * Get the current volume of the sounds controlled by this SoundControl. * @return the volume as a float. */ virtual float GetVolume() const=0; /** * Set the volume of the sounds controlled by this SoundControl. * @param vol the volume to be set. */ virtual void SetVolume(float vol)=0; /** * Unmute sounds controlled by this SoundControl. */ virtual void Unmute()=0; /** * Mute sounds controlled by this SoundControl. */ virtual void Mute()=0; /** * Get the current Toggle state. * @return true if sounds controlled by this SoundControl are activated, * false otherwise. */ virtual bool GetToggle() const=0; /** * Set the current Toggle state. * @param toggle true to activate the sounds controlled by this SoundControl, * false otherwise. */ virtual void SetToggle(bool toggle)=0; /** * Deactivate sounds controlled by this SoundControl. */ virtual void DeactivateToggle()=0; /** * Activate sounds controlled by this SoundControl. */ virtual void ActivateToggle()=0; };
Comments and differences with psSoundManager and SoundControl
- I still do not like the way PlaySound are handled here. I will find a better solution later.
- I have changed the name of Load(), Unload() and Reload() to the more explanatory LoadActiveSector(), UnloadActiveSector() and ReloadSectors().
- Note that the main SoundControl is treated differently as it controls the overall properties.
- The interface lets the user to handle more than a queue of sounds. Right now psSoundManager uses only one queue for NPCs' voices.
- Methods similar to psSoundManager::SetVoiceToggle() and psSoundManager::GetVoiceToggle() are not available since the user can access directly to the voice sound control through GetSndCtrl() (actually those methods are not needed even in psSoundManager).
- psSoundManager uses 4 psToggle. That class add to a boolean toggle only the callback feature that should not be accessed by external classes so I have decided to present an interface with only getters and setters. psToggle will still be used in the plugin module's implementation.
- Note that iSoundControl does not support the callback functionalities; indeed I do not think the user should use them. Those methods will be anyway implemented in the plugin module for internal uses.
- The only add to iSOundControl is the type.
Sounds in factories
In CS every object has a factory and multiple instances, so you can have a mesh factory, which is a rock, and then instanciate it multiple times by changing its coords, rotation and texture. It is possible now to associate a sound to factories and meshes; in this way it is easy to make a monster play sounds for example.
The ENTITY tag
A sound can be associated to a mesh factory or a mesh by inserting the tag <ENTITY /> in an xml document that describe an area (currently these documents are located in /art/soundlib.zip/areas). There are two types of entities: a factory entity and a mesh entity. The first one associates a sound to a factory, the second one to a mesh. Here there are two example that explain how to create an entity:
<ENTITY FACTORY="name_of_the_factory" STATE="1" RESOURCE="name_of_the_sound_to_play" PROBABILITY="0.5" MAX_RANGE="10.5">
<ENTITY MESH="name_of_the_mesh" STATE="1" STARTING_RESOURCE="name_of_the_sound_to_play_on_transition" PROBABILITY="0.5" MAX_RANGE="10.5">
In these two examples there are all the mandatory attributes of the ENTITY TAG:
- One and only one attribute between FACTORY and MESH that contains the name of the factory/mesh that you want to play the sound. If both those attributes are given, or if no one of them is specified, the tag is just ignored.
- One attribute STATE. The state is used to determine in which situation the factory (mesh) will play the sound. For example a factory called "rat" can be associated to a calm peep when it is in a normal state and to a more aggressive noise when it enters in combat. The states used by the client are those in the enum psModeMessage::playerMode.
- At least one attribute between RESOURCE and STARTING_RESOURCE. RESOURCE gives the name of the sound that must be played randomly during the period of time in which the factory (mesh) is in the given state. STARTING_RESOURCE is the name of the sound that must be played when the factory (mesh) enters the given state. STARTING_RESOURCE can be used for example to make a monster emit the last cry when it enters the DEAD state. If no one of these two attributes is given the tag is just ignored.
- One attribute PROBABILITY that gives the probability that the sound indicated by the attribute RESOURCE is played in a second. Note that the sound in STARTING_RESOURCE is always played at the beginning, no matter of the probability.
- One attribute MAX_RANGE that specifies the maximum distance at which the sound can be heard.
Optional attributes
There are other optional attributes that can be used in the tag to configure better an entity.
- MIN_RANGE: the distance at which the sound is heard with the highest volume.
- VOLUME: the general volume of the sound (it will vary with the distance).
- TIME_START: the time in hours when the factory (mesh) starts to play this sound.
- TIME_END: the time in hours when the factory (mesh) stops to play this sound.
- DELAY_AFTER: the interval of time in seconds during which the factory (mesh) cannot play any sound once the previous sound is finished.
A complete example with all the possible attribute for a factory entity:
<ENTITY FACTORY="name_of_the_factory" STATE="1" RESOURCE="name_of_the_sound_to_play" STARTING_RESOURCE="name_of_the_sound_to_play_on_transition" PROBABILITY="0.5" MAX_RANGE="10.5" MIN_RANGE="0.5" VOLUME="1.0" TIME_START="9" TIME_END="21" DELAY_AFTER="20">
for a mesh entity it is very similar but with the MESH attribute instead of FACTORY:
<ENTITY MESH="name_of_the_mesh" STATE="1" RESOURCE="name_of_the_sound_to_play" STARTING_RESOURCE="name_of_the_sound_to_play_on_transition" PROBABILITY="0.5" MAX_RANGE="10.5" MIN_RANGE="0.5" VOLUME="1.0" TIME_START="9" TIME_END="21" DELAY_AFTER="20">
The optional attributes if not given assume their default values that are:
- MIN_RANGE = 0.0;
- VOLUME = 1.0;
- TIME_START = 0;
- TIME_END = 24;
- DELAY_AFTER = 0;
The common sector
It is possible to create an xml document that define a sector called "common". The sound manager plugin treats this sector differently. You can use the common sector to describe the properties common to all sectors. At the current time the sound plugin manager supports only the entities of the common sectors. BACKGROUND, EMITTER and AMBIENT tags are just ignored.
Ambiguities and priorities
When a factory (mesh) entity is defined more than once for the same state, only the first one is picked up. All the other ones are ignored.
In summary the sound for a mesh can be specified in four ways: a mesh entity in its sector, a factory entity in its sector (that defines the sound also for all the others meshes produced by its factory), a mesh entity in the common sector and a factory entity in the common sector. These four ways have a different priority: the mesh entities in the current sector have the maximum priority and the factory entities in the common sector the minimum one. The complete priority order order is the following:
- mesh entity in its sector;
- factory entity in its sector;
- mesh entity in the common sector;
- factory entity in the common sector.
Distance lag effect of 3d emitters
Sound has a certain speed and openal unfortunately doesn't handle this factor so, even if it lowers the volume of distant sources, it will still make the sound of them run at the same time making the actual volume sound higher than it is actually. This would be about making emitters slow down when the player goes away from the source in order to unsync it from the other, more near emitters and go back to the normal speed after it has reached a certain offset from the more near object, in order to simulate this effect of distance and avoid equal sounds to stack.
Implementation notes
The sound's speed has been simulated in two different ways in the plugin:
- 3D sounds are played after a delay that depends on the distance between the source and the player. This one leans on the CS' event timer so it is not very precise but still it can be used to make differences between near sources and far away ones. Moreover this is the only effect that takes place when the player is not moving.
- 3D sounds are now subjected to Doppler effect. The source's speed is not taken into account to save a bit of time and memory. This means that the relative speed between an unmoving player and a moving monster is 0 m/s. This is not a big issue since this system is thought to unstack sounds and it's very unlikely that more than a moving source emits the same sound at the same time. This aspect takes care of changing the play rate speed of near sounds. It does the same for far away sounds but it cannot discriminate well between two adjacent sources far from the player.
The formulas to compute the delay and the change of frequency are:
- delay = d / SPEED_OF_SOUND
- heard_frequency = f0 * (SPEED_OF_SOUND - DOPPLER_FACTOR * v) / SPEED_OF_SOUND
where d is the distance between the player and the source, f0 is he emitted frequency and v is the speed of the player toward the source (positive if the player is going away from the source, negative in the other case). DOPPLER_FACTOR is just used to weight the contribution of the relative speed v.
Both the constants SPEED_OF_SOUND and DOPPLER_FACTOR are defined in psconst.h and are currently set to 331 and 2.5 respectively. To configure the effect of the distance lag effect you can change those values.
Not-Doppler emitters
It is possible to disable the Doppler effect directly in the xml file that define them with the optional parameter DOPPLER_ENABLED. Its default value is true. To disable the Doppler effect set it to false as in the example.
<EMITTER RESOURCE="blabla" ... DOPPLER_ENABLED="false" />
Overall code cleanup
Here will follow a list of the cleanup interventions in the code.
Sounds events
We need to allow to play a sound when some events happen.
Random sounds on monsters
When they stand still and idle. If the race is an intelligent race, then we can play some phrases, if not we can just have screams/sounds. These sounds can be played by using the factory sound system described above.
Sounds associated to behaviours/actions of monsters
Like he is angry and plays the angry sound. These sounds can be played by using the factory sound system described above.
Sounds for each weapon and attack
If you attack with a sword you should hear the sound of a sword.
Random sounds from environment
Buzzes of flies, water for rivers, wind and so on so forth.
Musical instruments
This will allow players to use musical instruments.
Sound part
This part implements the conversion between the musical sheet and the music that it represents. There are three major problems in this part:
- the song should be created from sound samples (the notes) stored in file of any format.
- There will be likely quite large musical sheets thus converting them into music all at once could be too much time consuming and lead to slowdowns of the client. Breaking down the song into pieces and play them one at a time is quite impossible to do because the event framework offered by CS is not enough precise to allow the right timing required to play the whole song. The only other solution is to update the sound stream of the song as the musical sheet is translated.
- treat a song as a normal sound in order to be able to use all the other implemented features.
There will be four main classes: SndSysSongData, SndSysSongStream, SongHandle and Instrument. These two will be likely be managed by a InstrumentsManager.
Instrument
This class represents a musical instrument. It keeps the decoded data of all the notes that it can plays and it provides this data when asked. Data of notes comes from the files that contain note samples. Instrument first converts the file into a iSndSysStream and reads the data from it. Since Instrument reads directly from the stream, the data will be already decoded thus the sample file format is not important.
All the instruments are defined in an XML document that InstrumentManager reads when created. The syntax of the XML language can be understood completely with this example:
<instruments> <instrument name="guitar" polyphony="3" volume="1.0" volume="1.0" min_dist="1.0" max_dist="50.0"> <note resource="E2" step="E" alter="0" octave="2"/> <note resource="A3" step="A" alter="0" octave="3"/> <note resource="B3" step="B" alter="0" octave="3"/> <note resource="C#4" step="C" alter="1" octave="4"/> <note resource="D#4" step="D" alter="1" octave="4"/> <note resource="E4" step="E" alter="0" octave="4"/> </instrument> </instruments>
<instruments> is the root. The tag <instrument> defines an instrument with its name, polyphony (i.e. the number of notes that it can plays at the same time), volume, the minimum distance where the sound is heard louder and the maximum distance at which it can be heard. <note> defines a note. The parameter resource indicates what's the name of the file that still need to be defined in soundlib.xml; step is the note's pitch; alter is the alteration of the note (-1 flat, 0 unaltered, 1 sharp); octave is the octave of the note (4 is the central octave in piano).
SndSysSongData and SndSysSongStream
SndSysSongData and SndSysSongStream implement respectively iSndSysData and iSndSysStream. It keeps the musical sheet and an instrument to play it with. When the renderer calls iSndSysStream::GetDataPointers(...) to fill its buffer, SongExecution generates the needed data dinamically by translating the sheet and using the instrument to retrieve the data of the notes. The sheet is thus translated just as needed each time.
SongHandle is children of SoundHandle. It is basically the same as its parent class but it uses SongExecution as its sound stream. This handle allows the song to be played as any other sounds with SoundSystemManager::Play3DSound(...). This means that the song are able to use all the already implemented features of other sounds.
Optional extensions
These things (or part of them) will be done during the GSoC coding time only time permitting. Otherwise they can still be included in the next GSoC or implemented by developers.
- Add ability to limit number of channels. The idea is to be able to limit the amount of sounds played at the same time coming from different sources. In the same areas you can have rivers, monsters, wind, and even the player clicking on the UI.
- Special actions to do when particular sound types are being played (for example reducing all other sounds/music volume)
- Allow voice speech in game through peer to peer connection.
- Have the walking sounds as a possible perception for monsters nearby. like a thief moving close to a monster to attack him first without making too much noise by walking, while a player with no sneak skills, will just make more noise while walking and the monster will hear this and attach him first.
- Have the sound of a player walking that vary based on the ground; so if it's rocky texture it's harder sound, and if it's grass is softer sound.
Skills needed
Our server and client are all coded in C++ and the sounds are defined in XML files. For this project C++ knowledge is needed, some basic knowledge of XML. No MySQL knowledge is required.
Difficulty
medium to difficult, depending on the parts of this which are picked up