ocarina/design.txt

===============================================================================
=                                                                             =
=                                 Ocarina 6.0                                 =
=                                                                             =
===============================================================================

My main goal for Ocarina 6.x is to plan out all of my actions before writing
code.  In the past I was adding features as I thought of them before thinking
out how everything works together, and this made Ocarina difficult to maintain
because I had no overall plan.  This document aims to fix that.

I will also create unit tests as I add features so bugs can be found faster.
Unit tests will be created for each module (file) in my backend library code.



Files:
	$HOME/.ocarina{-debug}/
		album.db
		artist.db
		genre.db
		groups.idx
		library.db
		playlists.lst
		track.db
	/usr/bin/
		ocarina
	/usr/lib/ocarina/
	ocarina/
		design.txt
	ocarina/gui/
		*
	ocarina/include/
		audio.h
		database.h
		database.hpp
		file.h
		filter.h
		groups.h
		idle.h
		idle.hpp
		index.h
		library.h
		playlist.h
		prefs.h
		print.h
		version.h
	ocarina/lib/
		audio.cpp
		database.cpp
		file.cpp
		filter.cpp
		groups.cpp
		idle.cpp
		index.cpp
		library.cpp
		playlist.cpp
		prefs.cpp



Install:
	Ocarina will be compiled into a single executable placed under
	/usr/bin/.  Any extra files needed to run will be placed under
	/usr/lib/ocarina/.



Printing: (include/print.h>
	Sometimes text needs to be printed to the screen so users (or debuggers)
	know what is going on.  Enabling dprint() when CONFIG_TEST is enabled
	means I will only need a single test.good file for output comparing.


API:
	void print(string fmt, ...)
		Print text to the screen.
	void dprint(string fmt, ...)
		Print text to the screen when debugging or testing is enabled.



Versioning: (include/version.h)
	This file contains a simple function for returning a string stating
	the current version.

API:
	const char *get_version();
		Returns a string describing the current version.



On-disk files: (lib/file.cpp)
	Data will be stored in the user's home directory according to the
	XDG / freedesktop.org specification.  This means storing data in
	$XDG_DATA_HOME/ocarina{-debug|-test}/ and storing configuration in
	$XDG_CONFIG_HOME/ocarina{-debug|-test}/.  In addition, I will support
	importing data from Ocarina 5.10 for conversion to the new format.

	In theory my file format will not change often, so it should be
	possible to use multiple Ocarina versions with the same data.  However,
	should the format need to change I will only support forward
	compatibility.  This means that downgrades will not be possible after
	a file format change.  To keep the code even cleaner, I will only
	support updating from the previous file format version.  This means
	that legacy support will be removed after the first file format change.

	Items should be written to a file with either a space or new line
	separating multiple values.

- Notation:
	File << aaaaa << bbbbb << endl is translated into "aaaaa bbbbb\n"

- File version:
	#define FILE_VERSION 0

- Hint where the file is located:
	enum FileLocHint {
		FILE_TYPE_CONFIG,
		FILE_TYPE_DATA,
		FILE_TYPE_LEGACY,
		FILE_TYPE_INVALID,
	}

- Open mode:
	enum OpenMode {
		OPEN_READ,
		OPEN_WRITE,
		NOT_OPEN,
	}

- File:
	class File : std::fstream {
	private:
		unsigned int version;
		OpenMode mode;
		FileLocHint hint;
		string filepath;

	public:
		File(string, FileLocHint);
		~File();
		const char *get_filepath();
		const unsigned int get_version();
		bool exists();
		bool open(OpenMode);
		bool close();
		string getline();
	}

- File format:
	File << FILE_VERSION << endl;
	File << <OTHER_DATA>;

- API:
	File : File(string filepath, FileLocHint hint)
		Resolve filepath to one of:
			XDG_{CONFIG|DATA}_HOME/ocarina/filepath
			XDG_{CONFIG|DATA}_HOME/ocarina-debug/filepath
			XDG_{CONFIG|DATA}_HOME/ocarina-test/filepath
			$HOME/.ocarina/
			$HOME/.ocarina-debug/

		If filepath is an empty string, set the file hint to
		FILE_TYPE_INVALID and do not set the filepath field.

	File : ~File()
		Close the file stream if it is open.

	const char *File : get_filepath()
		Return the full filepath to the file.

	const unsigned int File : get_version()
		Return the file version number.

	bool File : exists()
		Return true if the file exists in the filesystem.
		Return false otherwise.

	bool File : open(OpenMode mode)
		When opening a file for reading (mode == OPEN_READ),
			- Return false if the file does not exist
			- Open the file
			- Read in version from the start of the file
			- Return true

		When opening a file for writing (mode == OPEN_WRITE),
			- Return false if the file has FILE_TYPE_LEGACY set
			- Create missing directories as needed
			- Write version information to the start of the file
			- Return true

		Return false if hint == FILE_TYPE_INVALID.
		Return false if the file is already open.
		Return false if there are any other errors.

	bool File : close()
		Close a file after IO.

	string File : getline();
		Read an entire line from the file and return it to the caller.
		In theory a return value optimization will occur so returning
		a string by value won't be a problem.



Database: (lib/database.cpp)
	Ocarina 5.x created a different save file format for each type of
	data that needed to be stored (preferences, library paths, playlists).
	I intend to unify everything into a generic file format that can be
	accessed through a generic database interface.  The database code will
	be in charge of printing the "valid" bit for each DatabaseEntry so that
	child classes do not need to call into the parent class.  If valid ==
	true, the DatabaseEntry will be streamed out followed by a newline.  If
	valid == false the database will print the next entry in the vector.

	Modules should inherit from the DatabasEntry class and implement their
	own read() and write() functions.  The "valid" field will be stored
	before these functions are called, and the entry will be skipped if
	valid is set to false.

	The Database class is a templated class, so code could potentially
	get messy.  Normal class declarations can still exist in the file
	include/database.h and member functions can be written in the file
	include/database.hpp, which will be included by database.h.  Any
	function not relying on a template can be written in lib/database.cpp.

- DatabaseEntry:
	class DatabaseEntry { /* let database modify valid flag */
	public:
		bool valid;

		virtual void write(File &) = 0;
		virtual void read(File &) = 0;
		virtual void print() = 0;
	};

	File << <CHILD_CLASS_DATA>

- Flags:
	enum DatabaseFlags {
		DB_NORMAL,
		DB_UNIQUE,
	}

- Database:
	template <class T>
	class Database {
	private:
		vector<T> db;
		unsigned int _size;  /* Number of valid rows */
		File file;
		DatabaseFlags flags;
	public:
		Database::Database(filename, flags);
		void load();
		void save();
		void print();

		unsigned int insert(T);
		void delete(unsigned int);
		unsigned int size();
		unsigned int num_rows();

		unsigned int first();
		unsigned int last();
		unsigned int next();
		T &operator[](unsigned int);
	};

	File << db.size() << endl
	File << INDEX_0 << db[INDEX_0].valid << db[INDEX_0] << endl;
	File << INDEX_1 << db[INDEX_1].valid << db[INDEX_1] << endl;
	...

- API:
	Database :: Database(filename, flags);
		Initializes database to use ~/.ocarina{-debug}/filename.
		Set up flags.

	void Database :: load();
		Reads a saved database from disk.

	void Database :: save();
		Saves the database to disk.

	void Database :: print()
		This function exists only If CONFIG_DEBUG is enabled.
		Following a similar format for writing to disk, print the
		database to the console in a human-readable format.

	template <class T>
	unsigned int Database :: insert(T &);
		Adds a new item to the db, returns the id of the item.
		if DB_UNIQUE is set, check if the item already exists in the
		DB and return it's ID instead of adding a new item.

	void Database :: delete(unsigned int index);
		Mark db[index] as invalid (quick deletion).

	unsigned int Database :: size();
		Returns number of valid rows in the database.

	unsigned int Database :: num_rows();
		Return db.size().

	unsigned int Database :: first();
		Return the id to the first valid row or return db.size()
		if there are no valid rows.

	unsigned int Database :: last();
		Return the id of the last valid row or return db.size()
		if there are no valid rows.

	unsigned int Database :: next(unsigned int &id)
		Return the id of the next valid row or return db.size()
		if there are no remaining valid rows.

	template <class T>
	T &Database :: operator[unsigned int index]
		Return a reference to db[index].



Idle queue: (lib/idle.cpp)
	The idle queue is used to schedule tasks to run at a later time.  Idle
	tasks must inherit from the IdleBase class so that multiple templated
	IdleTask pointers can be placed on the same idle queue.

	Creating an idle queue in idle.hpp will create a new queue for every
	file that idle.h is included in.  I want to have a single, shared
	idle queue used by the entire application so to get around this the
	IdleBase class is used and implemented in lib/idle.cpp.

- IdleBase:
	class IdleBase {
	private:
		schedule();

	public:
		IdleBase();
		~IdleBase();
		virtual void run() = 0;
	};

- IdleTask:
	template <class T>
	class IdleTask : IdleBase {
	private:
		void (*func)(T &);
		T &data;

	public:
		IdleTask(void (*)(T &), T);
		void run();
	};

- Queue:
	queue<IdleBase *> idle_queue;
	float queued = 0.0
	float serviced = 0.0

- API:
	void IdleBase :: schedule();
		Add the idle task to the idle queue.  This should be called
		by the IdleTask constructor.
		queued++;

	template <class T>
	static inline void idle :: schedule(void (*)(T &), T);
		Create a new IdleTask to run the function later.

	bool idle :: run_task()
		If there are tasks on the queue:
			run the next task
			scheduled++
		If there are still tasks on the queue:
			return true
		else:
			queued = 0
			scheduled = 0
			return false

	float idle :: get_progress()
		Return (serviced / queued) to the caller.  If there are no
		tasks, return 1.0 to indicate that the queue is finished (and
		to avoid a divide-by-zero error).



Index: (lib/index.cpp)
	An inverted index allows me to map multiple values to a single key.
	Keys are tracked separately from the rest of the map so they can be
	found and iterated over without writing ugly code.

- Index:
	class Index {
	private:
		map<string, set<unsigned int>> index;
		set<string> keys;
		File file;
	public:
		Index::Index(filename);
		void load();
		void save();
		void print();

		void insert(key, unsigned int);
		void remove(key);
		void remove(key, unsigned int);

		const set<string>::iterator keys_begin();
		const set<string>::iterator keys_end();
		const set<unsigned int> &operator[](string);
	};

	File << keys.size() << endl;
	File << key << endl;
	File << map[key].size() << int_0 << int_1 << ... << int_n << endl;

- API:
	Index :: Index(filename);
		Initializes an index using ~/.ocarina{-debug}K/filename.  Pass
		an empty string if you do not want this index to be saved.

	void Index :: load();
		Reads data from a file.  Call after static initialization of
		Ocarina to ensure idle tasks are configured

	void Index :: save();
		Saves data to file

	void Index :: print();
		This function exists only if CONFIG_DEBUG is enabled.
		Following a similar format for writing to disk, print the
		index to the console in a human-readable format.

	void Index :: print_keys();
		This function exists only if CONFIG_DEBUG is enabled.
		Print the database keys to the console in a human-readable
		format.

	void Index :: insert(key, unsigned int);
		1) If key does not exist, create it.
		2) Add int to the set for the given key

	void Index :: remove(key);
		Remove a key from the index;

	void Index :: remove(key, unsigned int);
		1) Remove int from the set of values associated with key
		2) If the set is empty, remove the key

	const set<string>::iterator void Index :: keys_begin()
		Return an iterator pointing to the beginning of the keys set.

	const set<string>::iterator void Index :: keys_end()
		Return an iterator pointing to the end of the keys set.

	const set<unsigned int> &Index :: operator[](string key);
		Return the set associated with key



Filter: (lib/filter.cpp)
	Filtering is used to generate a subset of songs displayed by the UI to
	that users can choose from.  The inverted index is generated at startup
	so there is no need for a remove() function, since it will be wiped
	the next time the application starts.

- Index:
	Index filter_index("");
	map<string, string> lowercase_cache;
	unsigned int lowercase_cache_hits;

- Parsing:
	1) Convert the provided string into a list of words, using whitespace
	   and the following characters as delimiters: \/,;()_-~+"

	For each word:
	    2) Check the lowercase_cache to see if we have seen the word before,
	       a) If we have, return the stored string
	       b) Convert the string to lowercase and strip out remaining
	          special characters.  Add the result to the lowercase_cache;

- API:
	void filter :: add(string, track_id);
		Parse the string into substrings following the "Parsing"
		section (above).  Add each (substring, track_id) pair to the
		filter_index.

		To generate substrings, iterate over the word starting from
		the front.  For example: "dalek" would contain the substrings
		{d, da, dal, dale, dalek}.

	void filter :: search(string, set<track_id> &);
		Parse the string into substrings following the "Parsing"
		section (above).  We want to find track_ids that match ALL
		substrings, so take the intersection of all sets returned by
		the filter_index for a given substring.

	void filter :: print_cache_stats();
		Print cache hit and size information.

	void filter :: get_index();
		Return the index storing all the filter data.
		(Only available if -DCONFIG_TEST is set)



Groups: (lib/group.cpp)
	Groups are going to be a new feature in Ocarina 6 and can compare
	directly to Gmail-style labels.  Ocarina 6 will create dynamic groups
	that cannot be deleted by the user based on library status.  Similar
	to the library, groups should exist in their own namespace.

	In Ocarina 6.0, groups are a wrapper around a specific index.  Future
	releases will store user-defined groups in a file on disk.

- Index:
	Index group_idx()

- Default groups:
	All music
		All tracks are added to this group
	Library
	Banned Songs
		These groups are mutually exclusive.  A track is either
		in the Library or the Banned Songs group

- API
	void group :: add(name, track_id)
		group_idx.insert(name, track_id);
	void group :: del(name, track_id)
		grou_idx.delete(name, track_id)
	void void group :: list(list<string> &);
		return group_idx.keys();
	void group :: get_tracks(name):
		return group_idx[name]



Library: (lib/library.cpp)
	The library manages databases containing track information added by the
	user.  Ocarina 6 splits the library into multiple database tables for
	storing content.  The library will exist in a library namespace to
	to make functions and classes more unique.

	When a library : Track is created, it should be added to the "Library"
	group if it is NOT a member of the banned songs group.

- Album:
	class library :: Album : public DatabaseEntry {
	public:
		string name;
		short  year;
	};

	File << year << name

- Artist:
	class library :: Artist : public DatabaseEntry {
	public:
		string name;
	};

	File << name

- Genre:
	class library :: Genre : public DatabaseEntry {
	public:
		string name;
	};

	File << name

- Path:
	class library :: Path : public DatabaseEntry {
	public:
		string root_path;
		bool enabled;
	};

	File << enabled << root_path

- Track:
	class library :: Track : public DatabaseEntry {
	public:
		unsigned int artist_id;
		unsigned int album_id;
		unsigned int genre_id;
		unsigned int library_id;

		short track;
		short last_year;
		short last_month;
		short last_day;
		unsigned int play_count;
		unsigned int length;
		bool banned;
		string title;
		string length_str;
		string filepath;
	};

	File << artist_id << album_id << genre_id << library_id << track << last_year
	File << last_year << last_month << last_day << play_count << length << banned << endl
	File << title << endl;
	File << filepath << endl;

- Track: /* This struct lies outside the library namespace */
	struct Track {
		library :: Album   *album;
		library :: Artist  *artist;
		library :: Genre   *genre;
		library :: Library *library;
		library :: Track   *track;
	};

- Databases:
	Database<library :: Album>   album_db(album.db);
	Database<library :: Artist>  artist_db(artist.db);
	Database<library :: Genre>   genre_db(genre.db);
	Database<library :: Library> library_db(library.db);
	Database<library :: Track>   track_db(track.db);

- Updating algorithm:
	set<pair<lib_id, track_path>> known_tracks;

	1) For each track currently in the library, check if the track exists
	   in the filesystem.
	   1a) If the track does exist, add to the known_tracks map.
	   1b) Else, mark track invalid.
	2) For each file in the scan directory, check if (lib_id, track_path)
	   exists in the known_tracks map.
	   2a) If the file is in the map, do nothing.
	   2b) Else, add track to the library, to the groups "All Music" and
	       "Library", and then to the filter index.
	3) Save all databases

	The taglib library should be used for finding artist, album, etc. tags
	for each track.

	Use idle tasks for step 2 to break up tagging new files into chunks.
	This way the user will still be able to use Ocarina and scanning can
	happen while idle.

- Testing:
	A test library should be created to test adding tags and anything else.
	The command `arecord -d 10 </dev/zero | lame - -b 32 -h silence.mp3`
	will create a 10 second long, silent mp3 file.  Do something with this
	command and figure out how to apply tags!

- API
	library :: init();
		Initialize databases and read files from disk.  Fill out
		groups and prepare filter as tracks are read.
	library :: add_path(string dir);
		Add new row to paths table, update
	library :: del_path(unsigned int lib_id);
		Invalidate a path row and all tracks owned by that path
	library :: update_path(lib_id);
		Update the given library path, if valid.
	struct Track library :: resolve(track_id)
		Fill out a Track structure for the provided track_id
	const Database<library :: Album> &library :: get_albums();
		Return the album database.
	const Database<library :: Artist> &library :: get_artists();
		Return the artist database.
	const Database<library :: Genre> &library :: get_genres();
		Return the genre database.
	const Database<library :: Library> &library :: get_libraries();
		Return the library database.
	const Database<library :: Track> &library :: get_tracks();
		Return the track database.



Playlist: (lib/playlist.cpp)
	Playlists are a list of songs that the user has configured to play.  I
	will create a pool of playlists that will be filled by user actions.

	Playlists will be put on a "deck" that is used to give an order to the
	next songs played.  When deck :: next() is called, find the first
	playlist with PL_ENABLED set and call that playlists next() function.

	When a playlist is empty, remove it from the deck.

- Flags:
	enum playlist_flags {
		PL_ENABLED (1 << 0),
		PL_RANDOM  (1 << 1),
		PL_DRAIN   (1 << 2),
	};

- Playlist:
	class Playlist : public Database {
	private:
		database<track_id> tracks; /* Keyed on track id */
		unsigned int cur;
		unsigned int flags;
	public:
		Playlist();
		void add(vector<track_id> &);
		void del(vector<track_id> &);
		void set_flag(playlist_flags);
		const unsigned int get_flags();
		unsigned int size()

		File &operator<<(File &);
		File &operator>>(File &);

		void sort();
		void next();
	}

	File << flags << tracks.size() << tracks[0] << tracks[1] << ... << tracks[N];

- Deck:
	list<Playlist> deck;
	unsigned int current_track;

	File << current_track << deck.size() << endl;
	File << deck[0] << endl;
	File << deck[1] << endl;

- Deck:
	list<Playlist> deck;
	File << deck[0] << endl;
	File << deck[1] << endl;
	File << deck[N] << endl;

- API
	deck :: init();
		Read in the playlist file
	deck :: new();
		Adds a new playlist to the deck
	deck :: rm(N)
		Removes playlist N from the deck
	Playlist *deck :: get(N)
		Return playlist N from the deck
	deck :: next()
		Play the next song from the deck

- TODO										<<<<<
	What if each playlist has its own playlist_id for tracks?  This would
	allow for simpler removals, since I won't need to search for a track id.
	I can easily create a function for mapping a list of playlist_ids to
	track_ids...



Audio: (lib/audio.cpp)
	This file will introduce an "audio" namespace containing all of the
	functions interacting with gstreamer.  This will create a wrapper
	namespace that will be easier to work with than using raw gstreamer
	functions.

	The audio layer will also control the "pause after N tracks" feature
	so songs can be loaded without neeting to pass in a "begin playback"
	flag every time.

- Internal:
	Set up a message bus to look for end-of-stream and error messages so
	the next song can be played.  This function should call the play
	function after loading a track and after checking the "pause after N"
	count.

- API:
	audio :: init(argc, argv)
		Initialize the gstreamer layer and reload the track that was
		last loaded before shutdown.  Please only pass --gst-* options
		for argv.
	audio :: load_file(filepath)
		Loads a file path but does not begin playback.
	audio :: play()
		Begin playback
	audio :: pause()
		Pause playback
	audio :: seek_to(X)
		Seek to a position X seconds into the track
	audio :: stop()
		pause()
		seek_to(0)
	audio :: pause_after(N)
		Pause after N tracks, pass a negative number to disable.
	audio :: position()
		Return the number of seconds that the song has played
	audio :: duration()
		Return the duration of the current song in seconds



Preferences: (lib/prefs.cpp)
	Preferences make use of a special index where the set<int> is always
	size 1.  Preferences will be in the prefs namespace.

- Index:
	Index prefs(prefs.idx);

- API:
	prefs :: set(string, val);
		prefs.replace(string, val);
	prefs :: get(string)
		return prefs[string].begin()



Gui: (ocarina/*)
	The GUI will be written in C++ using gtkmm3 for (hopefully) cleaner code.

- Design requirements:
	- gtkmm3
	- Front-end for the library to add, remove and modify paths
	  - This can be hidden since it's not a common task
	- Double-click to play songs
	- Front-end for groups to add and remove tracks
	  - This is a common task and should not be hidden
	- Don't save / restore window size
	  - Some window managers already do this automatically
	  - Set reasonable defaults



Future work:
	I want to set reasonable expectations for Ocarina 6 so that I don't
	have to spend a large amount of time coding before releasing something
	to the wild.  This section will be a list of features that I want, but
	should be deferred to a future release so basic support can be coded.

	Hint:  If feature B depends on A, implement A in 6.x and B in 6.x+1

	- New default groups: (6.1)
		Unplayed tracks

	- Categories: (6.1)
		Use these to make "groups of groups" for better organization.
		Different categories can include Album, Artist and Genere
		dynamic groups in addition to user created groups (see below)

		The Artist, Album and Genre "tables" can be used to populate
		these categories.

	- User created song groups: (6.2)
		Basic add and remove features can be implemented using the
		Library and Banned Songs groups.  This will give me a chance
		to test saving groups on a small scale before letting users
		create anything they want.

	- Save a user's playlist as a group: (6.2)

	- Library defragment: (6.1)
		Ocarina 6.0 will leave holes in the library when tracks are
		deleted, potentially leading to fragmentation and larger-than-
		needed file sizes.  A "defragment" utility can be created to
		clean up unused slots.

		To help with fixing groups, a mapping of (old values) ->
		(new values) should be kept.

	- Fix track durations: (6.1)
		Some tracks in my library are tagged with the wrong duration,
		so fix them as they are played.

	- Track tag editor: (6.2)
		Make a pop-up window for editing the tags of a track.  Be sure
		to update the library information and the on-disk file.

	- Album art: (6.1)

	- Playlist custom sorting: (6.1)
		Click column headers to choos sort order
		Keep a list of fields that the user has selected and place new
		fields in the front of this list.  Use a recursive stable sort
		to do the sorting.

	- Better design file format: (6.1)
		Todo list in each document instead of all at once in the footer.
		Leave the Todo list out of the official "design document" and
		keep it in each individual section instead.
		XML?
		Code formatting?

	- Copy a song group to a different directory: (6.x)
		This can be useful for keeping an external device (like an
		Android phone) updated with the music you want to listen to.
		Complications: I have an mp3 mirror of all my music, and I
		want the mp3s to be synced.  Perhaps track mirrors in Ocarina?

	- Mirror directory: (6.x)
		I rip music to .flac, but keep an mp3 mirror directory to sync
		to other computers and phones.  An Ocarina tool to manage a
		COMPLETE library mirror might be a good idea so I no longer
		need to manage it externally.  This can still be done with a
		script, a cron job, and maybe a "mirror this track" option in
		the library?  Perhaps create a mirror group?

	- Extra testing ideas: (6.1)
		- Run tests through valgrind to help find memory leaks
		- Combine earlier tests into a single file